Powered by sharp

Welcome to lazy load.

My name is Lovell Fuller, freelance software developer, and also the creative maintainer of an open source library called 'sharp'.

Now for those who haven't heard of Sharp it's a image processing package, works with the NodeJS runtime and I released it about nine years ago and it's slowly worked its way into many big JavaScript frameworks, content management systems, a few CDNs use it.

What I'd like to do today is step back a little bit, take you on a bit of a journey so you can learn a bit more about how sharp came to be where it is today, and also potentially a little bit about what we plan to do with it in the future.

So we're gonna get back to 1782, 240 years ago.

Now, this is Élisabeth Louise Vigée Le Brun.

She was a late Rococo era painter.

She lived in Paris.

She mixed some of the the upper echelons of society at that time, and she became very good friends with Marie Antoinette, ended up painting lots of pictures of Marie Antoinette and, and other famous French people of that era.

This particular painting is a self portrait, a selfie if you will.

It currently hangs on display in the national gallery here in London.

Now, if you haven't been to national gallery, please do so.

Get yourself down there there's thousands of fabulous paintings and artwork so it's a really really good place to go.

It's in Trafalgar Square, in London, please do get along to see it.

Now, why is that important?

Well, there's lots of other painitings obviously in the national gallery, we're looking here at constable's Hayway.

And what happens with these paintings is that they're quite old-in many cases, hundreds of years old,and over time, paint starts to degrade.

And it's very important when you're doing art preservation, to work out where paintings re startingto degrade, what colors are starting to go, where you might need to do repair work, where you might need to do more restoration.

And so 30 years ago in 1989, a team, a small team got together at the national gallery to work out what they might be able to do to make this art preservation process a little bit easier.

And they created this.

This is a multi-spectral camera uh, it scans seven different channels.

So you've got red, green, blue, you've got ultraviolet, also got three infrared channels.

It can scan pictures up to about 10,000 by 10,000 pixels.

This is over 30 years ago, right?

This is, this is the most advanced technology at the time.

So this camera pumping out very, very large images, the software at the time, wasn't able to handle it.

So the team at National Gallery, led by John Cupits and Kurt Martinez had to create their own software from scratch to be able to process these such huge images which didn't ft inmmory at the time, they barely fit on disk at the time.

And what they created was the VASARI, image processing system.

So for VASARI is named after Georgio Vasari, he was an Italian art historian.

But there's a bit of a backronym for VASARI.

So I think in the end it became the visual arts system for archiving and retrieval of image processing system.

What a snappy name.

Luckily they thought better of calling it that, and instead of just to simply call it 'VIPS'.

VIPS Is a combination of a libraries processed images, and also a UI as well to display them.

And so what we now, what was them VIPs we now know better as libvips, and libvips is the software upon which Sharp is built.

Now the reason the libvips is used by Sharp and actually lots of other bits of software now as well, is that it's very fast.

Why is it so fast?

Well, it takes a slightly different approach to move processing than other libraries.

So for example, image magic.

Yes.

It has a thread pool.

What it does is it takes each stage of the processing, assigns all the threads to that stage.

libvips prints horizontally across the system and allows the IO system to control the threading.

So in this picture here, you've actually got effectively four threads.

One thread's doing decoding.

One thread is doing encoding and the data sync, and you've got two threads, which are each running, all of the operations.

What this means is that each thread is given a small region of the image to work on.

And then it applies different operations to that region image.

This means that that region of the image is very likely to stay within the CPU cache, which makes it much faster because you're not suffering from CPU cache evictions, And so generally the throughput is significantly higher than other processing libraries.

Typically it's anywhere from about three to five times faster than image magic, it does, depending on the operations you perform.

Typically in generally it is much, much faster.

Now libvips wasn't just used at national gallery here in London.

It has also been used at the Uffizi Gallery.

This is in Florence in Italy of course.

I'm looking at the the Birth of Venus, a Botticelli painting.

I believe that this has being used to help do some restoration work on this painting.

I think the third gallery to start using to libvips was the Louvre, in Paris one of the people who worked in London, at the national gallery, moved to Paris to go and work there.

So they now use lbvips as well for their art restoration projects such as the Mona Lisa, the very famous painting.

In addition to the Rijksmuseum in Amsterdam also uses libvips.

And know this is a particularly interesting painting.

This is Rembrandt's Nightwatch.

Very, very famous painting indeed.

If you go onto the, the Rijksmuseum website, there's a deep zoom image of of this painting , which allows you to go to an extraordinary level of detail, each individual brushstroke.

Now this [?] image was made possible thanks to the libvips.

But in addition, the, the team at the Rijksmuseum have now be using machine learning to try and predict parts of the painting that are now missing.

So sort of the paint's very old, it's cracked off.

I believe some of the edges, were actually removed at one point to reframe it a couple of hundred years ago.

And so the team there have been using machine learning to predict what, what the missing paint should look like.

And it's only because of the ability to use software, such as libvips they've been able to do this.

Please dogoandlooka the Rijksmuseum website, it's fabulous.

We're going to fast forward a little bit now.

2011.

I start working for a company called Net-a-Porter in West London.

They're based above the Westfield shopping center in Shepherd's Bush.

This is the the office-it's kind of, it looks a bit like something from Gatica the film.

And it was kind of this strange kind of collision of fashion, of technology, of content, of commerce.

And the company had a very sort of strict control about everything that it did.

And that included.

its own photography studios.

So this picture here is of the studio.

It's about 10 meters away from the previous space we looked at.

So every single item that was sold by Net-a-porter, remember this is all luxury fashion items, all of those photographs were taken in the house, in the studio, they were manually edited in-house.

And the output at the end of this was a massive TIFF file of each picture.

And so of course this was 11, 12 years ago.

And what we did was we had a shelf space, which ran, it was magic and converted that massive TIFF file the few different [?] images for each parts before it then got published to a shared server somewhere and then was presented on the web.

Couple of years later, 2013, this is when responsive web design is the *big* thing.

Everyone's rushing to sort of make their own website responsive so it works on various screen sizes, various mobile devices, and Net-a-Porter, we're kind of caught short.

We've got a few dimensions, for all our thousands of products, but actually what we need then is these other dimensions to fit on this particular new iPhone that's just come out.

And we were a little bit stuck-we can't really do that because we've got these 1000 images, and this image magic script is providing those but we want other dimensions.

And I thought, well, maybe there's some way of doing it.

So I kind of went home one night.

This was August, 2013 and I looked around and I'd been using NodeJS for a couple of years at this point.

I was aware of libvips at this point, but I hadn't ever really used it before.

And I kind of just sat at home one evening after work and threw some bits of code together.

I'll doing some JavaScripts and I'd obviously done some C++ before.

So it was able to kind of marry those two languages together.

And I published version 0.0.1 of Sharp.

It only dealt with JPEG images is pretty minimalist, but it kind of works.

And actually we thought maybe we could use this in the office.

But obviously Net-a-porter weren't the only company using Sharp.

Lots of other things started to happen too.

So here's a picture of the Westlink in West London.

Probably 10 minutes walk from the Net-a-Porter office.

And there's this beautiful coloured artwork under the bridge, alongside the canal.

Now the word 'colour' is has a certain spelling in British English, it's got a u in it and I suspect in other variants of the English language they also had a 'u' in it.

In Australian, Canadians use 'u' in word colour, but our friends in the USA decided to remove that 'u' from the use of the word color.

And if you work on the Web, everything you do that involves the word colour is always going to be C O L O R, even if in your head, you spell it C O L O U R.

Now what I wanted to do with Sharp was to make an API that was usable by lots of people.

And so here's an example.

This is a screenshot from the Sharp documentation.

I wanted to make sure that people could spell English however they wanted to.

So if you want to convert the color space of image, you spell color, the way that he wants to spell it, you can spell it with a U or you can spell it without a U.

Both of them work.

Now, I've had quite a lot of people contact me on the internet, privately, email, whatever, saying this is such a good feature.

They love the fact that they don't have to use American spelling.

It actually makes them stop and think about the API.

And actually that's kind of important to me.

I think it's been really important to focus on this level of detail about the API.

People like these touches.

And I think that that helps with it's adoption.

So also in 2014 was the first time that Sharp really start getting noticed by the people.

In May, I think it was, 2014 it kind of broke into the top 10 most starred repositories om Github and maybe like a week or so later, I think it finally actually got to number one.

There was someone in the US in Silicon valley, someone fairly well know tweeted about it, and then suddenly lots of people were starring it.

I mean, starring is one thing.

Right.

But actually what you want people to be doing is using it.

This is a this is an English two pound coin.

Don't know if anyone's ever seen one of these before, I've got one here.

If you have looked at this coin, right, you noticed around the side of the call, this has some writing.

Now you won't be able to see this but what it says is "Standing on the shoulders of giants".

Now, this quote is attributed to Sir Isaac Newton.

But I suspect, probably people have said this kind of thing a long, long time before he came along.

But the reason I mentioned this is because Sharp very much stands on the shoulders of giants.

It's not just [?], It's not just Sharp/libvips-there's a whole other host of libraries that are involved.

The current version of Sharp, when you type in "NPM install Sharp", downloads a binary, A a tarball of binaries, and these are 28 of the dependencies, which you get when you download Sharp.

These are all C, C++, Rust libraries.

It's quite a spread of libraries.

It's amazing how much different things that you can probably tell from some of the names what these things might be doing.

Lot's of stuff around text, all different image loaders, encoders.

But having such a large number of dependencies, obviously you start to worry about security vulnerabilities.

Luckily for the last four or five years, libvips has been part of the OSS-Fuzz projects.

For those who don't know, this is where random images or random inputs, are kind of pulled out of thin air, and some servers in Google, I think there's now about 25,000 servers, try and generate as many random input files as possible to test all the various code paths in the library.

The idea being that if you find these really weird edge case inputs you can try and squash some bugs.

So actually we, this has been running for a few years now for libvips we have found a few bugs, luckily, nothing too major in libvips itself.

However, because libvips has all these other dependencies, we're also effectively doing security testing for all the libraries on which libvips depends.

We have found a couple of slightly more serious bugs.

One of them was granted a CVE that actually resulted in a patch because it was also dependency, chromes.

Chrome had to patch based on bug that the libvips OSS-Fuzz fuzzes found.

But OSS can't find everything.

A big German company adopted Sharp a few years ago and they were a little bit worried about security.

And so they commissioned their own pen testing.

Doing basically black box testing.

So they've got an outside security consultants test it.

And what they discovered was that it was possible with a very well-crafted PNG image to view parts of the memory of the server it was running upon.

So what we can see here is the top half of the image is kind of acorrupted PNG.

And the bottom half of image, is a different image that somebody else submitted to their service.

So what this is enabling people to do is look at other people's images, whichyou really don't want it to do.

So luckily the company that found this did a very responsible dislosure to myself and John, the lead maintainer of libvips, and we were able to within, I think about two days, get new versions of sharp out the door, which basically prevented this kind of attack from happening in the first place.

And then we made, shored that up with some changes to libvips as well, so that no one else would suffer from this.

So I was going to say say if you find security things like this, do disclose them responsibly.

And we can act at least, hopefully all the problems [?], fuzz is finding the edges.

If we introduce a small bug into like the main line of the code, within 48 hours, OSSFuzz has told us there's a bug.

So it's really really good for [?] So moving on-2017, Now this is an art exhibition, it's taken in Italy I think it was about a year ago.

And it's a photographer called Angelica Dass.

Now she's a Brazilian lady and she travels the world, taking portraits of people from all walks of life, all races everywhere.

There's such a wide spread of, range of people as possible.

What's special about what she does is that when she presents the portraits, she includes the background as the skin tone of the person whose photo she's taken.

You'll notice that it says Pantone, she includes the Pantone colors.

Now why am I showing you this?

Well, one of the features that was added to Sharp andf back into libvips eventually was the ability to do smart crop.

So, this is why you take an image for one aspect ratio and not only resize, but also crop it to a different aspect ratio, and in doing so you have to throw some of that image away.

And so what smartcrop allows you to do is to select the most appropriate part of the image and libvips and Sharp now have something called the 'attention strategy' for doing a smart crop.

And so this looks like things like edges.

So, where there's a lots of change in the image it looks for areas of high saturation.

But thirdly, it also looks for skin tones.

Now there have been some attempts to do skin tone detection by some companies which have got them into trouble, particularly using machine learning where the data set hasn't been very good.

But we didn't want to run into that problem with libvips.

So instead we used the 'Humanæ' project, some images from it, and all of the colors, not the luminosity, just the colors just the Chromacity values of all of these backgrounds, all of these photos.

And so when you smart crop in Sharp today, it's looking for chromacity valleys, which have appeared in this Humanæ project.

So hopefully we're not gonna run into the same problems by doing this, that a lot of other smart crop implementations have.

So how did Sharp, this is how Sharp came to be, but how did it come to stay?

Why is it still with us?

Well, in 2018, Sharp was adopted by Gatsby.

Gatsby is kind of the protagonist of the JAMstack era.

It was one of the first big static site generators to use React.

Still going very strong today, they've made, the team that look after Gatsby are doing such a good job still of of improving it all the time.

But Gatsby have been using Sharp almost since their first release for all of the image processing that it does.

That's probably how most people have come to use Sharp in some way, because they've tried to install Gatsby, and the installation of Sharp has failed.

I'll come on to that in a minute.

Moving on 2019.

So in 2019, there were 19 million downloads of Sharp.

So this was the first year where I really started noticing, wait a minute, this is, this is almost like increments...

exponential growth in the number of people using Sharp.

It's a good idea of where things have come since then.

Last year, 2021, there were over 75 million downloads of Sharp from NPM the trends hasn't stopped this year so far, we currently the download from NPM is approximately three times per second.

So right now, three times per second, somebody downloads Sharp from NPM I think we're on course to do something like 90-95 million downloads this year, which is just absolutely mindblowing just phenomenal the level of usage.

2020.

So this was the year when people start talking about other image formats, newer image formats.

The iPhone, I believe, started to support HEIC, H E I C, formats this year.

Now HEIC is an interesting one, because it's, it's a bit of a patent minefield, right?

There's lots of patents around HEIC and HEVC.

And to counteract that the association for Media, AOM, had created the video project called AV1.

Uh, And some people at Netflix decided to take that a step further and create a static image format out of the video CODEC.

That is what we now know as AVIF.

So in 2020 Sharp added support for AVIF, actually we added support for something called libheif to libvips in order to be able to explain this AVIF behavior to Sharp.

And I believe now if you, if you go on the web, if you're served and AVIF image it, it's very, very likely that has actually come from Sharp.

I don't know what the [chances?] are, it's something big, almost half of the AVIF images on the Web tat people are generating are now using Sharp which is again all a bit mind blowing to me.

Last year in Sharp.

One of the things which quite a few people have complained about in the past is about the installation size of Sharp.

We saw that there's 28 different C libraries and more behind the scenes and that has over time, started to grow and accumulate.

So at the start of 2021, start of last year, I thought "what can we do is try and make things a bit better?" So some very simple things to do.

We can just change the way that those those libraries are compiled We've been looking at do we actually need all these dependencies?

Is there some, is there like some flagging in the complier that removes unnecessary data?

I made a couple of small commits to libvips itself to improve some of the color profile handling to reduce its it's footprint.

So now I think when you, when you type "in NPM install Sharp" it gets typically about a seven megabyte download between six and seven megabyte depending on the platform.

And then when that's decompressed onto disc, it takes around about 20 megabytes.

So that's the idea I'd like to now keep it at about that level.

So if we were to add new libraries in the future, we might have to look at some other ways to reduce it, just to try and keep it a little bit more constant, but I think you can expect future versions to be around about this level.

Also in 2021 I realized that a lot of people were using Sharp alongside tools like imagemin, and I believe imagemin is now no longer being maintained.

So, wouldn't it be good if some of the features that people were using from imagemin might be made available in Sharp itself?

One of those is mozjpeg which is a fork of libjpeg turbo.

Actually, this is very easy to drop in because they're almost like for like replacing each other so now if you wanted to use the features of moz jpeg, you just have to add mozjpeg: true, to your code.

But if you don't specify that, then the behavior is exactly the same as always used to be uh, still uses the old libjpeg turbo path.

So you can choose whether you want performance or whether you want the more optimal file size.

Under the tools, part of the imagemin family is pngquant.

Again, this does quantization of PNG images.

So it turns full color into a palleted image, which usually takes a much less disk space, though it is a lossy operation.

And if you want to take advantage of this now in Sharp, it's not quite the same as pngquant, but it's very, very similar.

You just say you want pallete: true on PNG and the output should be much more optimal in terms of file size.

Now Gatsby was the start, but lots of other JavaScript frameworks were also starting to pick up and use Sharp.

Next JS is an interesting one.

So the NextJS added Sharp as a dependency and then they removed it as a dependency.

And I don't know if more people complained because it was added or more people complained cause it's taken away because now it's an optional dependency.

So if you have a next project repo and you haven't already done some, make sure you add Sharp to your list of dependencies and next will automatically use it for all the image processing.

You'll find that it's probably around three to four times faster than the built-in library that it uses.

For those who use Vue, Nuxt.js also has Sharp under the hood for all of its image processing by default 11ty, fantastic tool also has Sharp buiilt-in.

The new version of the Parcel bundler has Sharp built-in for all of its image processing.

There are other tools as well that have, that use Sharp not necessarily built-in but as plugins things like Vite, [?], but it seems to be an increasing number of JavaScript libraries and tooling that does use it.

It's not just the JavaScript libraries.

AWS recommend the use of Sharp for any image processing using Lambda Sharp now, where else, it works, not only in Lambda, the old Intel, it also works on the new Graviton 2R, 64 ships as well.

So they, they provide us with like, it's a template for you to be able to create a Lambda issue that does each process.

This is from the AWS Docs.

So we're almost up to date now, 2022.

That's this year.

Still new features being added.

Probably the most requested feature of Sharp is the ability to resize animated gifts.

And so now with the latest release, you can do this.

There's an example of the code you might have to use.

It's took a little while to get in because the support, or actually the GIF encoding library support wasn't very good anywhere I would say.

It was quite difficult to find a really good GIF encoder.

And so about six months ago, I was approached by a team at Shopify to improve the gift handling in libvips.

And I worked with a German student called Daniel who had created just a couple of weeks before that a library called cgif.

We managed to get it into lipvips and therefore now exposed to Sharp itself.

Shopify has now, actually have now recruited Daniel to go and work for them.

So not only did he actually get his opensource software used by everyone, he's also got himself a job out of the deal as well.

That's what opensource is meant to be about.

Making those connections, getting people [?], such a good outcome for him.

So that's kind of where we are now, with Sharp.

But what about the future or at least the near future?

I don't like to predict too much.

So I mentioned earlier lots of people, when they use tech they complain, people use Gatsby, they complain about Sharp cause it didn't install properly, there are errors at instill time, and it has been difficult.

I've been trying to improve the installation experience for people in the Sharp, as much as I possibly can but without support from the package managers, it's always going to be a bit of a battle.

Now that a lot more people are using native dependencies, for example, ESBuild is very popular these days.

That's all native code, suddenly a lot more interested in actually improving support for basic modules, and providing prebuilt binaries.

If you con...

Experience with the Python ecosystem that has the concept of PYWheels, this is what I think is hope hopefully is what's going to be happening with this new proposal from NPM.

So there can be a concept of package distributions.

So this is an example of what I'm probably going to have to add to the Sharp package JSON.

I plan to publish binaries for the libvips and Sharp to NPM.

When hopefully, when this new version of NPM comes out and supports this, it means that we don't, all those install, time problems go away.

I think Yarn is also planning to adapt to some of those things as well.

So this is, this is going to make all the difference in terms of installation time.

It, this, this is a big thing, this is goingto be good, really looking forward to this, this extra feature of NPM.

Also coming up in the near future is support for the JPEG XL format.

And this is a very modern take on the JPEG format.

It doesn't really represent the JPEG format at all any more.

And it's kind of, it's almost ready to go.

I'm saying I would hope that later this year, the, the libjxl library is going to be ready for us to ship, start to ship in Sharp.

libvips is already integrated with it.

There are some missing features particularly around metadata, but I'm hoping that within the next, I would say the next six months, we should have JPEG XL support shipping in Sharp.

Which is very exciting.

Which actually then is the sort of thing which allows people to start to use it in browsers.

Cause it's, it's kind of like a chicken and egg thing right?

The browsers support it then no one can create the images then are people are going to start shipping images in that format?

So I'm hoping that by having, doing it in sharp people start to use this format.

WebAssembly lots of people want to use Sharp in a browser.

I keep getting issues opened on Githubabout this and really, sorry.

It requires a NodeJS runtime.

It's not going to work in the browser.

However, we have a, one of the contributors to, maintainers of libvips [?] has been doing some amazing work, getting libvips and some of its dependencies to cross compile down to WebAssembly.

I'm not quite sure it's ready for prime time just yet.

It's quite a few kinks to iron out, but this is something I think as soon as it's in a state where we can start use it, I would like to start trying to provide a version of Sharp which will work in browsers, and that will work in some of the edge compute services that are available, which maybe don't have a nodeJS run time, but do have a JavaScript runtime.

Things like Cloudflare Workers, Deno, that kind of thing.

It's not there yet.

Don't get too excited, but I'd love for this to be a thing.

In terms of sustainability, of libvips.

libvips Itself has been around for 30 years.

It's kind of slowly grown in popularity, certainly over the last five years, a lot more services are usng it.

We have an open collective, we've so far had donations just over 30,000 US dollars so thank you very much to you, if you've donated.

We we have had lots of small individuals donatingto this.

Also there are some corporate donators too-Google donated 10,000 US dollars Shopify Donated 10,000, US dollars, Gatsby donated, it just under 5,000 US dollars.

This makes all the difference because it allows us to pay interns to do work on libvips.

to help improve it.

We have [?] to fix documentation.

But we also used the money to help some of these smaller dependencies on which libvips builds.

And I mentioned there were 28 dependencies, that ship with Sharp.

Some of those are big [?] that are backed by big corporations.

They've got loads of money.

There'll be all right.

They can look after the longterm.

Some of the dependencies are a little bit smaller, often they're maintained by just one person.

And so we've used some of the money that we've received from into the libvips open collecting to pay other maintainers.

We had, [?] Kleis, I mentioned earlier he did the work on Web Assembly.

We've donated to libspng G, which is a modern PNG, encoder and decoder.

Just about to give a thousand dollars to Marty who looks after LittleCMS, we use this for the color management color profile support.

Marty's very kindly offered to donate his money to a mental health Charity, thank you Marty.

We've got a thousand pounds, as I told you a thousand dollars going to Dirk.

He looks after libheif.

Now libheif It has had a corporate backing up until now, but it sounds like the corporation that has been backing it may not want to maintain it anymore.

So we're hopefully going to be working with Dirk, helping him get something off the ground so that libheif itself, can be sustained in the future.

Because it's very important we use it in Sharp for example.

I mentioned cgif earlier, we gave Daniel some money and actually Danielk now got a job off the back of this so well done.

Also we've given some money recently to a few charities in Ukraine who've been affected by the attacks from Russia.

So thank you for listening.

Thank you to anybody who has contributed to Sharp.

There are over 150 of you now, who've contributed new features, bug fixes.

Even if it's just fixing documentation.

Thank you.

That's so helpful.

Really helps everyone else who uses it.

Thank you to the, I dunno, they must have about 3000 people who've opened issues on the Sharp repos on Github.

It's anyone asking questions that actually helps me understand how people start to use Sharp and then that allows me and other people to improve the documentation which then helps people too.

So thank you very much for that.

If you'd like to get hold of me, I'm available all the usual social media places.

My username is lovell.

You can find me on Twitter, on Github, and I'm generally quite easy to find.

Thank you very much for your time today.