How To Be a Web AV Artist

Ruth John at Pixel Pioneers 2019
Sign up for Conffab Free to watch this and hundreds of other videos for free
[Music] you [Applause] regulators but you can't be any geek off the street gotta be handy with the steel you know what I mean Risto oh goodness me one day I'm gonna stop doing this talk and I'm gonna be way less nervous because the whole thing is written in JavaScript and full disclosure anything can go wrong at any time okay hey Bristol hi my name is Ray I'm from the UK I'm from the southwest I'm from like near these towns called Bristol Anbar so this is kind of really cool because I get to talk Aristo it's amazing I haven't travelled very far I don't got off of a flight it is so good and I see so many familiar faces in the audience as well like it is actually a pleasure speaker thank you so much for inviting me Oliver um so yes my name is Ruth this is the part of the talk where everybody tells you who they are and what they do and why they get to stand on a stage and tell you about all this stuff you just saw it so I'm going to go through and just talk about that right I am here to talk to you about how to be a web AV artist what yeah so I struggle with the practicality of this talk quite a lot because web AV artist doesn't really make a lot of sense a V stands for audio visualization that's what you just saw it's visualizations with audio doesn't really belong on the web right you're not going to go back into work on Monday morning and just code some audio visualizations into your website or that idea T I do I do struggle with the practicality at the store I don't know whether you saw this Venn diagram it did come out a couple of years ago but I took this very literally on the left hand side is what you what people talk about in conference talks and the right-hand side is what you should talk about in conference talks so I have taken this very literally and I have made two lists to make sure that this talk is useful for you right so things I have to avoid as per this diagram analytics operational excellence chat ops and orchestration I think I can manage that but I do have to get in duct tape workarounds bike sharing and coffee right okay now I've got my talk anxiety out the way we can learn about where baby and what is an AV artist a B stands for audio visualization this is more visualization than it is about audio really because it's taking the audio and making visualizations based on the sound all right so we're going to be doing a little bit more visual than we are audio but they're both in there I have control over volume down a little bit as well I quite like the audience that I'm seeing now because I was worried it would be excruciating me lunch young he remembers this thank you very much audience sometimes I do that every site what for those of you don't know this is winner this is a piece of software which you install on your computer and it plays mp3 files mp3 files are sound files they're actual files you're not streaming them right so you have a ton of these files you put them in this piece of software and it plays them it's very lightweight it's very cool this is how we used to play music when we first got computers that's not entirely but this I remember this this is how we used to play music but this wasn't the best bit about winner right this was the best bit of winner it had a visualization engine ya see some of you yes I remember so you used to play your mp3 and other sound files are available through winner and you used to fire this up and this went in time to the music that you are playing so you sort of play music and then sort of watch this we didn't have Netflix back then thank you in a visualization engine this for me I was doing audio visualizations back then as well this is it sort of at uni sirs almost 20 years ago and this must have been the biggest influence of people who are doing it back then I think I think I'm okay to say that another word for these audio visualization people is Vijay by the way we describe ourselves as Vijay so if you hear me use that term I'm just talking about the same thing um let's see how we go about building this with the technologies that we already know right we know web technologies that's what we're familiar with so well sort of I'm sad that that's kind of ended so quickly I keep wanting some replay and replay about over seven years ago we started to get some really interesting api's in the browser that made me stop and think could I do this audio visualization stuff could I do this visualization like VJ stuff in a browser environment it may not have been my smartest moment but it was fun trying right so the first thing that we need to be able to do audio visualizations is audio we have a Web Audio API this came out 2011 2012 and we we can use it to get audio so the first thing that we need is audio there's four main ways with the Web Audio API you can create to your own audio so you can create an oscillator and this is going to create your own sound probably not what we're looking for we're looking for Sam that other people are playing and you can grab it from the dom so if you've got an audio or a video element in your HTML you can grab the audio source from that there is a method to do that with the Web Audio API you can stream it from your device so you can stream it by the microphone or the camera now this is quite useful so you can basically just pipe in the audio that is coming in from your device that's pretty cool or you can just go and get a file so if you've got a file sitting on a server you can go and fetch it you can bring it into the audio API I can you can use that sound so they're your four main ways so that's great this is a sound file oh okay so I'm picking up on the sound file being imported into the audio API this one which is my preferred method is just picking up on my microphone on my device alright and this is really good because you don't know what situation you're going to be in you don't know who which DJ you're going to play with or which the talk you to be doing right so picking up on the mic it's a really interesting way of doing it because it means I doesn't matter what ambient sound is around so that's my preferred method now that we've got audio we need to analyze that audio I wasn't allowed to talk about analytics ok we've got a whole talk to go we've got a whole talk to go so we need to be able to analyze this sound to be able to move the visuals in accordance to what the sound is playing and we can do that with the Web Audio API as well the Web Audio API is kind of really really big it's kind of a little bit intimidating if you do want to talk it like research it more if you are a music file I'm here you can talk to me afterwards there are some really really good mdn Doc's and tutorials on mdn about it as well but there is an analyzer node which is the thing that I need to be able to analyze the audio there's lots and lots of these nodes in the Web Audio API that's how it all kind of stuck together so you have like the way that you bring in audio as a node whether that is via a file or via the Dom the way that you modify it the way that you can analyze it you can put filters on it you can change the volume these are all nodes as well and you kind of string them all together and that's the way the API works so we have an analyzer node this is the thing I want I really got my sound source now I'm going to attach and analyze a node to it to be able to get that data back and so that's what this is doing and what that returns for us is an array we can create this array ourselves we can make it as big or as small as we want and it's basically giving us the volume we're gain or the amplitude of any sounds going from low sounds to high sounds at any given time so if you're playing something quite bass te will get numbers here so these are going to be louder so we're gonna get higher numbers on the low sounds in the array and if we play something quite high we're gonna get numbers up around here in the higher numbers of the array alright we're going to this in a bit more detail in a minute but that's basically what I'm dealing with I'm dealing with an array of values essentially this is my analysis and this is what this is really representing all I've done is I've created an array I've created an element for each item in that array and I'm just changing the height based on the number that I'm getting back that's it that took me about yeah okay don't treat quite that long but I kind of fit html/css simple wasn't actually quite that simple and I do sort of have this thing about people saying things are simple but that is what this is doing I'm getting back this frequency data and I'm just looping over it and just playing an eye element in my HTML and just changing the height of that with CSS this was one of my first experiment and this was actually quite tricky back in the day because we didn't have that streaming of the mic the way that we have it now with the Web Audio API that all came out of sort of doing things like this so it has become easier which is nice the thing is I did this and I said okay we can do that what else can we do I should probably do some experimentation around this concept and as we all know if you want to do some experimentation you need some inspiration this is Bridget Riley she's a super cool artist it says so on the slide so true and she does these huge paintings of shape like they're really really big they're bigger than this screen and I was like I really want to make these into visuals that we really cool so this is my first inspiration of just chucking things at the DOM and making them move like this this is actually weirdly just a whole bunch of items positioned with flexbox and the width is just changing based on that data then getting that that's it a little bit tanky but it works all right and for those familiar with CSS this is the CSS triangle border hack that you can do you just use border stylings to make a little triangle and I'm just changing the opacity based on those levels that I'm getting back so if I shut up goes away yeah oh and then what happened was I was like okay I could do this maybe I could try some other technologies which I don't get to try in my day job remember this is a couple of years ago transforms so CSS transforms were quite prominent because I felt like I'm gonna be moving stuff around I probably want to rotate stuff we're talking about visuals here I thought could I take everything that is in the screen make it a quarter of the screen and duplicate it around the screen there's not a nice symmetry stuff yeah you can do that so this is actually just one thing going on here and I just really function to duplicate it and right pace it around that's it and I thought well if we can do that can we drop a Giffen yep Thank You flicker there it is you can drop a given and so on and so forth this it kind of doesn't stop something of coming out around the time words blend modes and filters in CSS filters are very cool so filters change things like black and white for images they can blur images you can do hero tapes that actually works on video as well which for a visualization artist is great you can just drop a video in the DOM and invert it and do cool stuff blends blends a different modes when you're layering different Dom elements up and they actually work on SPG's so this is a visualization I did for Jess Khan for you two years ago we just came back off this year's which is basically this on steroids this is it now I've sort of moved over to SVG which I'll get to in a minute but this is all like layered on top with blend modes because they all work with SVG as well another really nice thing to try especially when you're doing this kind of visual stuff one really prominent thing with custom properties CSS custom properties otherwise known as CSS variables now this meant that I could sort of power my CSS from my JavaScript which was very cool because I was doing a lot of stuff with the CSS but I was also kind of hacking it with JavaScript because the data was coming into JavaScript but I kept this meant I could actually just pipe it in directly and this is how I did it so this is my CSS up here I created a custom property called level on the root I've just arbitrarily called it one and I'm just using it to sort of change some properties like width and height so you can make them bigger and smaller and then where I'm getting my frequency data where I'm getting these volume values in my JavaScript I'm aria dressing this custom property so I'm resetting it to the data that I'm getting in so it's updating my CSS in real time that's JavaScript or life when your CSS people welcome to Houdini's this is like a couple of years ago before we started on the Big C so this document.documentelement style set property level is doing that and I'm just changing that number one two something else and so I was doing all of these things all of them right and you might notice so this is reflecting or rotating around and changing the width oh hi I'm doing some custom properties I've got some blend modes going on here it started to get quite janky it didn't really like operating very well I wasn't allowed to talk about operational excellence see that this is going let's go back to this this wasn't very good and actually this is a gift this isn't all the other demos are actually really in these slides because if I put this within the torque structure it just breaks I think it really cannot cope with all the stuff that's going on so it's thinking a lot about performance this is Ben this is my friend Ben and this is an extension sort of rubber ducky it's not just I have got this bug if you can find somebody who you can talk to your weird ideas about they're your friend talk to your weird ideas about them he was extremely helpful I was doing an interactive installation I was just talking it over with him and we were chatting wasn't nice to approach at ops I was talking to Ben and he made a few good points he was saying you're working with the Dom you're now working with SVG is because SVG's just give me sort of more shapes and more openness and it's not just sort of squares and I elements and things like that anymore and you're working with data so I'm working with the analysis data and you're doing a lot of maths have you thought about using d3.js d3.js for those who don't know it's sort of a chart based library it does the pie charts and bar charts people are doing incredible stuff with it now Noddy was here a couple years years ago talking about the stuff that she does with it but it's a data visualization library basically and he was right I am visualizing data so I'm not somebody who jumps on board libraries too quickly especially when I'm doing personal projects like this because it's a learning experience so if somebody says to me there's a library for that I'll be like I'm gonna write it which I'm not saying is a great idea by the way but it is good for learning so this is what d3 gives us so I did a whole bunch of experiments around d3 because it did actually help improve performance this is a book called super graphic by Tim Liang and it's really really nice it's just kind of data visualizations of comics and graphic novels and you can just flick through it if you're into it it's just it's all just really really a beautiful book just to flick through and so speaking to this and wait for the photo and I thought this is really good inspiration for my g3 set of visualizations and these then started so this is just a sunburst chart basically and it's just oh go quiet and then I go loud you can see how that would work pretty well music there's things like the concentric circles you can do just really really easily this is reimagining that spectrum that we saw at the beginning of the talk I'm just using circles on a diagonal instead of up-and-down squares and things like flowers so it starting to work in circles now there's a lot more trick going on but it's d3 kind of makes it easy it really does and this SVG is look a lot cleaner and prettier and yet it works better d3 is actually doing the same thing that I was doing before which is creating a shape per item of the array so I'm taking that analysis array that I had chucking it 3d 3d trees like yeah I'm going to create an item per shape this is fine but I wasn't satisfied with this purely because of the analysis that the Web Audio API does now it's still doing the analysis well it's still doing this what we call fast Fourier transform on my sound that means that it's picking out all the frequencies that I want but it's not doing it in a way that I think creates good visualizations bear with me so this is our array there are those sounds at the bottom and naught Hertz that's very very very low that's probably the lowest that you could hear if you could do it that low the sounds at the top of the array this is what we're actually analyzing here a 20,000 kilohertz now your ear can hear that but in reality for the most part it actually doesn't so there's not a lot of activity here I'll show you here this is a keyboard I'm sure we've all seen a musical keyboard not your computer keyboard it sits on this frequency chart around here so music that we listen to every day it's not going to go up here you'll be like it could it could but it's probably not and actually if you look at the numbers coming out of an array like this this whole top half usually comes out at 0 just because there's no activity so I don't need to this data it's nice that it's there but I don't need it I want this data right the other thing about this data is it doesn't really represent music this is our keyboard I highlight here three notes on this keyboard they're all of note a they're different octaves so what that means is that they all harmonize really really well together ones lower ones in the middle ones higher right I'm pretty sure we've all played on keyboards and you understand it there's a middle key here this runs a 440 Hertz right so on that spectrum we still got our Hertz this one's 440 this one above it is 880 Hertz that's a gap of 440 Hertz cool bear in mind that these gaps here are equal we would assume that that's 440 Hertz lower but it's not that's 220 Hertz that low key so that's a two hundred and twenty Hertz gap so these gaps are actually getting bigger even though the note gaps are not the frequency gaps are this isn't represented in the array that I'm getting back from the Web Audio API which is fine for what we need it for but when I'm sort of all right like visualizing audio I kind of want it to be a little bit more even so I want to compensate for what's going on here so slice off this bit of the data don't need it and spread the rest of the data out simple Matt took a long time I haven't actually looked at this for about two years and I'm like yeah we could definitely refactor that now I'm not gonna walk you through that it's basically what that is trying to do which processing what is interesting I've got a before-and-after this is before right you can see I don't know what this is lousy okay you can see how there's a lot of activity around here so this is the bottom of that array that you saw I think you saw where the numbers are and there's nothing going on up here this is after I've done all that work everything's a little bit more even we're getting more of a representation in fact that this song is higher right I think that makes the best visual it doesn't even stop there I just wanted to bring that in because it's after lunch and I thought it made you think at the moment I still AM I was still working with SPG's and actually when you start talking about a hundred shapes like a hundred SVG zero Dom it doesn't like that either so very quickly changed over to a 2d canvas I've been doing some work around a 3d canvas which is the visualization you saw at the top of this talk this is actually a 2d canvas and I'm just drawing these shapes it's exactly the same sort of thing I don't use d3 anymore I have actually built my own system to do this and this means I can just chuck however many there are on there and this isn't running at 60 frames per second it's running at about 25 but for the purposes of displaying this in the cloud I think it's pretty good I mean if this is the browser I haven't pulled anything off the main thread yet either okay so we've got visuals that's great but as a VJ when you are actually doing this you need a way to control your visuals now on our computers we have keyboards I could just punch in a couple of like keystrokes and get each visual to change when I pressed a different key that's fine I did some experiments with the gamepad API there's a gamepad API in the browser so taking a gamepad and just seeing if that would work to sort of change up the visuals but it's a little bit flaky all game pads are different so there's nothing it's just a bit hard alright settled on MIDI I know you're thinking MIDI just quick rundown it's not that bleep e sounds that you think it is MIDI is a data protocol right this oh thank you this the keyboard we saw this is a MIDI enabled instrument so this is a MIDI instrument it has onboard audio when you play it it plays you sounds but most keyboards these days come with MIDI capabilities if I plug this into my computer it's going to send it MIDI signals which is just data it's just I am a keyboard and I'm playing this note some instruments have velocity so I'm playing this note this hard right this came about in like the late seventies early eighties because they had a rise of electronic instruments synthesizers drum machines all that kind of stuff and they would take them into recording studios and then they had no way to talk to each other it actually had all these instruments no way to orchestrate with each other right orchestrated I've got there is a really instrument you can also get MIDI controllers so these send MIDI signals but they don't have any onboard audio they're a little bit more affordable all right you've probably seen these this is a launchpad I haven't screened now and these are far more common when you're working with a computer because they are more affordable they're more lightweight you can use your computer for the audio that's kind of the idea you press the button you make the computer make the sound no not me we have a web MIDI API so in the browser just like we have a Web Audio API we have a web MIDI API and this basically picks up on a MIDI signals that are coming into the browser right don't you like that this is my handbag MIDI controller this is a minim they all are very similar you usually get something different sizes they will have buttons on they'll probably have dials on and you can just run this code make sure your browser is listening to this let's see whether it's directly in with it and you can press buttons and I can change those visuals and I can change different parts of the visuals as well the system that I have running now you can change things like the colors you can do inverting you can do all those effects on your actual MIDI controller so with any luck this is still connected so I have a little mixer down the side this is a continuous controller so normally this would be a dial but because I've got this special little Bluetooth controller it's just a little slider down here and then you can set up the buttons but you get the idea right and then you can set these up however you want it's almost part of the build to be honest is this real this is just a game this is very odd and building a piece of what should be OS level software with a bunch of JavaScript in the browser with a bunch of sort of api's that people are thinking why do we have this why do we have like MIDI in a browser I get asked this question actually a lot because this is also I do this I do code I make websites how do you do this how do you get into this what is this this is really what we're focusing on when we talk about digital arts right otherwise known as creative coding this is it I sort of came up with an idea and did some experimentation this is sort of now become a thing and we're not web is not the only place where this is really really prominent and this is becoming more and more popular every single popular coding language has some sort of creative coding environment right so c plus plus you've got cinder and open frameworks java has got processing you've probably heard of that one that's probably one of the most popular that's what most of the artists use Ruby has got agile and sonic pie it's the gel is new it came out last year I don't have a chance to look at it yet but you can see those sort of consistent thing going on here we on the web the word my favorite thing is we have this browser which comes with a whole bunch of stuff without having to install anything all right we've got those API is there's so many more api's it got mentioned we've got ambient light we've got accelerometers we've got geolocation just out of that box that's pretty cool there are JavaScript libraries there's things like p5.js you were 3j s there's tone j s for audio which are amazing and should be used but they are built all on top of this technology this web technology that we so took me like the other extensions to this is when you're doing this sort of creative coding or this digital art you're looking for data as well I'm looking for audio data a lot of artists use data from you as a human or data calls like weather data or Twitter dates or things like that all still so easily done in the browser so is this realized it's just a game this is a project that I made here I took this audio visualization software made into a little pack so I have my laptop my MIDI controller there's tiny little pocket projector and put it underneath this little pack and sort of walked around it with an usherette tray this is actually me projecting onto the streets of Bristol a couple of years ago so this is with the actual pack just in a browser over the laptop I wrote an article about it and I'll show it in the article a couple of people came back to me and said we're doing this too okay well not the only one and what happened posture is we've got together and we started doing this stuff together because that's what you do in the PFC you find like-minded people and you share ideas and you do this stuff together so as Phil mentioned we have live jf it's a collective there's an extra member now there's musicians there's people who build music software in the browser with JavaScript there are lighting artists there are people that are controlling massive lighting breaks just buy this whole MIDI thing and turning it into DM X which is what lighting music and of course visualization artists and hardware as well you can do neopixel type like wearable things and all the rest of it so that's what we do every so often when we get together and we share ideas and we do it we play games or this whole stage it's just running off web technology this is great we have a github organization so it's got to be real right there is also a slack channel if you go on our website and you are interested in this please join our slack town because we talked about this sort of stuff I'm running a little early but I don't think it matters because I think we were running a little bit like there is one more thing I do have to address obviously I'm sure you're aware and the keen-eyed among do you might have noticed all right this whole thing is held together with duct tape right Ben is on a bike there's whole function to work around and I am drinking coffee thank you very much you