Building amazing apps with Project Fugu APIs

Thomas Steiner at Summit '22

Transcript
Accessible Slides

Cool.

Thank you very much.

Yeah, so I'm Tom.

You can find me as Tomiac on Twitter or [email protected] on Masterdon.

And yeah, I'm gonna talk about building amazing apps with Project Fugu APIs.

But first I wanna start with a story.

Have you been there?

You browse the Web and browser Web, and you think oh, that's a really nice looking website.

For example, you browse the Web Directions website.

And you're like I wish I had a vector version of this image.

And you use your dev tools and you inspect and oh, it's Web Directions org event, blah, blah, blah, blah, blah.

Something something something square dot jpeg.

And on desktop it looks a little different.

If you compare the two images, they do some art direction here and interestingly, if you inspect on desktop, it already is an SVG.

So based on the view port, if it's desktop ish, it is an SVG.

If it's mobile ish, it is a jpeg.

So let's just for the sake of the moment, work with the JPEG version here.

If I isolate this, if I open it in a separate tab, you can see this is how it looks like and it's full glory.

And I mean that seriously.

So I did not know that this artwork would be the subject of the keynote.

We all know the story of it now, which is even more amazing.

And yeah, so let's see if we can vectorize this image, which is a jpeg.

And the usual solution people flock to and me included, is like we use a search engine, for example, Google or Duck Duck Go or Bing or whatever, and you type in something like convert JPEG to SVG and then you click the first result and it happens to be Convert io in this case.

And I have to say, I'm not meaning to pick on them or anything, it's just the literally first result that pops up and you see some ads and yeah, you wonder should I upload my images here because in this case it's artwork.

It's not even mine or, if it's, I dunno, a photo of my, kids, would they use it for training I don't know any kind of AI and use it in face scanning in the future?

I don't know.

But in this case I'm like, yeah, okay, let's do it.

And you upload it and you can see it starts to do its thing and it becomes green and it shows different ads and more ads, and you start to be more excited, and then you press the button and it starts converting and yes, This is gonna be good.

There's gonna be good.

And you wait and wait.

And then it says, conversion completed.

And you're a hundred percent, 110% excited.

You click the download button.

And then you open it in a wait, there is scroll bars, but it's all white.

So what's going on there?

And then you scroll a little and oh, It actually did successfully convert the happy face from the Luna Park.

It's yeah, there is something, so it's an upside down smiley.

So yeah, it's not quite, but okay.

Yes, a little bit maybe.

And just to be sure, I actually inspected and looked if there is more, and there is like I could highlight the eye and stuff, but a lot of the information is lost.

But there's got to be a better way.

because a lot of these tools that you can find online, they all rely on a tool called Potrace It's a command line tool that that was developed by Peter Salinger a couple years ago.

And yeah, it just works pretty well in, in theory and what I did is I just built my own tool.

It's called SVG Code.

I did something smart with it with a domain name.

I had a dot de domain name that was free.

So I registered svgco.de, which spells SVG code.

And yeah, you can see it succeeded in converting this image pretty well.

It's also available in en-AU and in dark mode.

Sorry, someone had to crack that joke.

It's actually implemented, if you go to SVG code.de and you select en-AU you can see it upside down.

Sorry, someone had to, so yeah, it is available in en-GB if someone wants to do an en-AU , I'm happy for a pull request.

But yeah, if you switch to en-GB , you get your behaviour, colour and 'ou' stuff and stuff.

It's also obviously available, in en-US, but anyways, it's available in dark mode and so on.

And yeah, just wanna talk a bit what SVG code does differently than the tool that you may have seen online?

First, it's an offline enabled PWA, so it runs entirely on your client.

There's no Server to trust apart from obviously like the first download.

But then you can inspect the code.

It's all open source, obviously.

There's no upload.

There's no data that leaves your computer at all.

And the way it does color is by poster posterization.

That's a hard word to say.

And you can see it a little bit here in this snake picture from Wikipedia.

So the upper one is in full color, and then the lower one is with reduced color.

So posterization is essentially reducing the colors in the image.

And the tool also exposes all of Potrace's expert options.

So you can fine tune and tweak the the results.

And when you save the SVG, it uses a tool called SVGO that optimizes SVG and makes it smaller.

So these are just a couple of the things that SVG code does differently.

As I said, it's an installable, offline enabled PWA, and the way this works is with something called Vite plugin PWA.

So I'm using Vite with the vanilla JS template.

So no react, no anything.

It's really just vanilla JavaScript and it uses Workbox.js under the hood for making all the PWA-ness, so the installability and offline enabled-ness and so on.

But because of the Vite plugin PWA, I didn't even have to touch the service worker code.

If you wanna build a PWA without dealing with this kind of low level stuff I highly recommend Vite plugin PWA.

So Project Fugu.

What is Project Fugu?

It's an effort to close gaps in the Web's capabilities and by that enabling new classes of applications to run on the Web.

And I was asked, what about the name?

So there's his famous Simpsons episode where Homer eats Fugu fish and things he might be poisoned.

And he thinks he has only one day to to, live.

And With some of these APIs, it's a little bit like Fugu.

If you cut Fugu it's amazing.

It's a delicacy, but if you cut it wrong, it can be deadly.

And some of the things, some of the APIs that we have in Fugu, they can be deadly.

Air quotes of course because like you have file system access as an example, and if you give and Web a Web app, if you give a Web application access to your etc/passwords file on Linux for example, this could be not so great.

You need to hold these APIs right, and put also safeguards and fences in place so that people don't harm themselves when they use these APIs.

So it's a, tongue in cheek name for a project.

And yeah, project Fugu, the blowfish here is the official logo and yeah, the target is building amazing apps with Fugu APIs.

So I betta mention some.

And the first API that I wanted wanna mention is the window controls overlay API.

And you can see it on the left-hand side.

That's the application once you have installed it.

And then you can click that little chevron here.

So this little triangle thing, the Chevron.

And then you can see how the title bar area becomes useful for content.

So you can put your content there, put a menu there, and you can see there's this little progress bar that shows when the application is busy.

So you can, use the title bar area.

That would be pretty useless in general because you installed the PWA so you know already that you are in this application.

So the title bar is superfluous.

So yeah, by using the window controls overlay, you can make that title bar actually display useful information.

How does it work?

So essentially there's a new API on the navigator app object called navigator dot windowControlsOverlay.

and on the windowControlsOverlay, you can then have an event listener that listens for the geometrychange event.

So whenever the geometry of the window changes, so when you click the Chevron this event is being triggered.

And then you can just launch this function here and you can see it's just checking if navigator.windowControlsOverlay is visible, yes or no.

And if it's visible, then it just toggles the class sets of the menu and of the main.

So if I go back my menu is this thing here in the title and the main is just this left hand bar.

So it just literally removes and adds classes so that they get displayed in this position or in that position.

So super low super low tech works really easy.

All you need to do is listen for the geometrychange event and then do something with CSS classes and check this property if it's visible or not.

So that's windowControlsOverlay aPI.

The next API is a little bit more dangerous a little bit more fugu.

So file system access.

As I said, if you upload the wrong kind of file or upload is the wrong term already.

If you open the wrong type of file yeah, you are in trouble.

But the file system access API has some safeguards in place.

For example, you cannot even access your system files, it's just if you try to, it'll just block you and say, this is not allowed.

Anyway, I'm using the browser-fs-access ponyfill library that I've built that allows me to use the file system access API when it's available and else it falls back to the regular input type file.

And the a with the download attribute element and what is cool about this API is also you get a FileSystemFileHandle when you open a file.

And this FileSystemFileHandle is something that you can then persist to indexedDB.

And if you do that, it allows you to, so you can see here on the right hand side, you open a file and then if you reload the application, It asks you do you want to continue where you left off?

So it checks if in indexedDB there's a FileSystemFileHandle and you can make a flow where it would check.

Then is there this file handle do you want to continue where you left off?

And this is also integrated with the drag and drop API.

So if you go to your Windows Explorer or Mac OS finder.

You can drag any kind of file onto the application and then you get this FileSystemFileHandle and you can then directly edit the file in the application So how does it work?

Let me show you at the example of the drag and drop API.

So you have on your document a Drop Listener.

So most of this code is really just pretty standard.

So you get the event.dataTransfer.Items.

So first you need to prevent default of course, and then you get the items, you get the first one and you check if the first item is a file.

And if it's a file, you can load the image.

So this is just application logic here, or you open the image and then you get the file handle by calling item.getAsFileSystemHandle.

And this file system handle is then something that you can store in IndexedDB.

So I'm using a library here that makes this a little bit easier.

I just set the file handle to this handle variable, then I get here.

So super easy.

There's no magic.

It's really just working with the drag and drop API and this one new addition to the API that then gives you the FileSystemFileHandle that you can then, persist to indexedDB.

The next API that I wanna talk about is the async clipboard API.

So you can see here on my screenshot I have my finder on Mac and I'm on my image downloads folder here.

I right click and I say copy.

So I copy a file to the clipboard and if I then go into the application, I can paste the file directly into the app and work from there with the image.

So let me show you how this works, but first I wanna talk about the other way around.

So if you copy something and paste something of course you wanna get data in, but you also wanna get data out.

So what I can do here is you can vectorize this image and then copy the SVG by clicking your copy SVG button, and then you can paste the image data into another application to then, for example, fine tune the paths here.

So this is boxy SVG, another PWA that allows me then to modify the paths that Potrace has vectorized here.

So if I want to, I can then just copy and paste into other applications right from my own application.

So how does copying work?

The cool thing about copying is it is multi mime-type.

So what does that mean?

SVG is an image format, but SVG is also a textual format.

So you can just look at the source code if you want.

It's just XML.

So what it can do is you can just onto the clipboard, put both representations.

So you can put the textual, representation or you can put the image and if you then paste, for example, into into VS Code it'll be the source code.

But if you paste into something like Boxy SVG the image editor you can then use the image representation.

So how does it work?

Essentially you get from the SVG output, the SVG code then you, optimize it with SVGO to make it smaller and optimized of of course.

And then it basically just consists of creating two blobs.

So one is of type text/plain, that's for the source.

. And the other is of type image/svg+xml.

So that's the image representation.

So you have two blocks.

One is a text, one is an image, and then you write them onto the clipboard by calling navigator.clipboard.write.

Then you have an array, and then you have a new clipboard item.

And this clipboard item has an object as a parameter, which has as the keys, the blobs types.

Text/plain and image/svg+xml.

And then as the value the actual blobs.

and that's it.

That's how you can put stuff onto the clipboard and work with the clipboard.

And by that, make your application open to any kind of other application that can deal with the system clipboard.

So could be Adobe Vector, illustrator, blah, whatever.

It could be Boxy SVG, it could be VS Code, it could be a text editor.

Anything that works and integrates with the clipboard.

It can be Web based or it can be just a platform specific application.

The next API that I wanna call out is the file handling API.

And you can see the screenshot is pretty similar, but this time there's a small difference.

So right click on an image and then I go open with, and you can see very down here there is SVGcode.app.

So what is happening here?

I installed the PWA and Mac OS now learned that SVGCode can handle image files.

So it offers itself as a file handler for image files.

Pretty cool.

Imagine that it's an PWA that you have installed from the Web and it integrates super well with the operating system.

How does this work?

So here you have a new object on the window, which is the launchQueue.

And the launchQueue gets consumer.

So you set a consumer that then gets launchParams.

So you can see the launchParams here.

And then you can just check if launchParams.Files is an array of length, something else than zero.

And if it's something of array length other than zero, you can then get the handle and the rest is pretty much the same as before.

So just some application logic.

You obtain the blob URL and so on.

That's really just how it works.

The secret is actually just the window launchQueue which allows you to get any kind of file that was passed to the application during the launch.

Next API: Webshare.

I show this on Edge, so you can see here there's the Edge icon.

So what you can do is if you have the SVGCode app open in Edge, you can click on share SVG, and then the Windows in this case, Windows 11 share Widget would open and yeah, you can see I haven't really used Windows much, so there's no other applications, but I can share it to my internal Outlook something, address that Windows has created for me.

So this is really just for demo purposes, but you can see Windows takes over.

So if you have applications installed on Windows that can handle images, they would offer themselves here.

So you could share, for example to an image editing application, or it could share to, a an email application like in this case an email address.

And by that, integrate directly with the operating system.

This works on mobile as well.

So sharing on mobile is something that is pretty natural but it's also starting to appear on desktop more.

It's not yet implemented in Chrome which is why I show Edge.

So how does it work?

You have the share button that has a click listener.

You get the SVG code, you optimize it you obtain a suggestedFileName, and then you prepare a new file that is yeah, essentially just a blob a glorified blob.

And you create this object called data with a files property.

And then you put in an array of this one file.

And then you check on the navigator, can the navigator share the data?

So this is just a safety check is the navigator in theory, capable of doing so.

So is the format supported, for example.

And if everything goes well, you can then navigator.Share the data and the rest is then just the operating system taking over.

So Windows in the complete case or iOS or whatever that would then deal with showing the share sheet or showing a share widget, whatever.

And making sure that you can go from there and share with the operating system.

The inverse is the Web Share Target API.

So if you have any kind of image open, you can then share it and SVGcode will offer itself as a share target.

So you can see SVGcode appears here now.

And this is the Windows paint application, which is of course capable of creating images.

And if I have created an image there, I can share it to this application and open it in SVGCode for further processing.

How does this work?

PWAs have a Webapp manifest, and in this Webapp manifest you have a new property called share_target, where you tell the application or the operating system rather what kind of files your your application can deal with.

So here you can see, I can accept images of type jpeg, png, webp, gif.

I could add avif I just didn't add it because the slide would be too long.

So any kind of image formats that you want to have.

The rest is just really just a form here.

So you have a form submit url.

You have a method multipart form data, sdn coding type.

And then the parameters is just this files array.

And, the service worker.

So this is the only service worker code that I had to write.

You have a fetch listener and then you check if the fetch event request URL ends with share-target.

So you can see this was my Action url, so share-target and if the share-target URL ends with this and if the method is POST, so POST you can see is the method that I had to use here.

Then I just continue with processing the information here, so I extract from the fetch event, the form data.

and the rest is really just dealing with forms.

So it's no difference from regular Web forms.

You get the image, then you have the image, and yeah, then you can use the rest to process the image locally and make sure that it gets over to the application.

So you can see here working with the Cache API because this is in the service worker, so you need to get it over to the main application.

And the way to do this is by using the Cache API, and then I just redirect the response to the main URL with just this little parameter here to mark that this is coming from a Web share target.

This is SVGcode and you've seen in this in this application alone, I've used 1, 2, 3, 4, 5, 6.

So six different Fugu APIs.

Window Controls Overlay API, File Handling API, Web sharing, File System Access API, Webshare Target API, the Asynch Clipboard API.

So six APIs used in one application alone.

But of course there's a lot more.

So we have a project, Fugu API Showcase, and of course SVGCode is part of it where you can submit your applications that you build with Fugu APIs and you can filter by APIs that you are interested in.

So for example, if you're interested in other applications that make use of, let's say, Window Controls, you can filter in the Fugu API showcase and see all the applications that use this particular API.

You can also, of course, search for applications.

So if you wanna search for SVGCode and see it in the showcase, you can do that.

The URL is hidden in this QR code, so if you wanna scan the QR code you can do that now.

And yeah, this is the API showcase, but yeah, Fugu being a project that is cross companies.

So we work with Edge, so with the Microsoft folks we work with Intel, there's Electron there's Google obviously.

So there's a couple of smaller other companies, Samsung, for example, is Samsung is not a small company.

So with, other companies and Samsung and other companies.

So, did this make sense?

Anyways, so we work with other companies on this project, so it's not just Google, that's what I'm gonna say.

And everything is happening in the open.

So there's the Fugu API tracker.

I can see all the APIs that have shipped that are in development that are behind the feature flags so far.

And yeah, you can also see the QR code for this showcase.

And yeah, this was the talk, if you're interested in the slides, scan this QR code you can follow along.

As I said, I'm tomayac on Twitter.

I also posted the link to the application there and the the slides again.

And if you want, you can ask questions.

I can also quickly show you the application live.

I guess we have how much time left?

Five minutes.

Let's see.

So anyone has a question or should we just show the application live?

So I can quickly just jump out of the app.

So if you go to svgco.de, you can see that's the application running in a tab.

But of course I can also run it as a standalone app.

So you can see that here.

Window Controls Overlay, you can turn that on and off so you can see how the application changes the different view views here, I can open an image image here.

Let's see.

I can just SVG myself so you can see this is me as SVG.

You can see this is running on Chrome now.

So on Chrome, I don't have the Web Share Target API, so it's not being displayed in the menu.

So it's using progressive enhancement already.

I can go to the Finder.

Let's first close the app so it's not open anymore.

Now let's take the Fugu fish.

I can right click and say I wanna open this with SVGcode here it is.

And you can see the application launches.

This is really hard to SVG to, to vectorize because it, it has this gradient.

So it's super hard to yeah, get this right.

So I can see here that's the gradient that I'm talking about.

What else?

I can copy the SVG and paste it into something else where is it?

SVGOMG, for example, by my colleague Jake Archibald.

Second go there and you can see it has a paste markup field.

So here it would use the textual representations.

If it pasted in here, you can see it's now in the application, has done some optimizations.

That's that.

I do have Windows running here, so let's see if this is working.

I can open paint.

It's a little slow because it's emulated of course.

I can draw anything.

I can then say I wanna Is it send?

Yeah, so it it, it's called send here, which triggers the share.

I can then click SVG code and hopefully.

So you can see this was very fine.

Probably not a great example because it's just literally one pixel, but you've seen it opened the thing.

Is it visible at all?

No, but let's, just try again.

Let me paint some more.

Ooh.

Spraying spray.

Here we go.

And let's use color [audience] the wrong track.

Okay, let's see.

I can then send this.

SVGCode pops up.

Ah, it's not great.

Let's see.

Oh, here we go.

Yay.

Yeah, so some of the settings I had triggered some of of course I can show you that so I can reduce the number of colors it doesn't really make sense for this.

I can change how it should be speckled or not.

So increase the quality, decrease the quality, whatever.

We can see that here maybe a little bit.

So if you have it less speckled, you can see that some of the speckles disappear and so on.

So yeah, progressive enhancement, it's working on Windows.

Because Windows has these APIs, it does not work in Chrome, cause Chrome doesn't have these API yet.

And with that, thank you.

Have you been there?

speech bubble og Thomas saying "That's a nice-looking website!"

Screenshot of mobile version of the Summit website

Slide from before, caption now reads 'wish I had a vector version of this image!-an arrow points to the URL of the header image.

The image open in a image editor. Caption now reads 'So yeah, let's see if we can vectorize it!'

The usual solution

Screenshot of google results for JPG to SVG. Caption is now a smiley emoji

Screenshot of Covertio JPG to SVG converter

Screenshot of the Converter in action

Convertio now says 'converting your file'.

Convertio now says 'Conversion Completed!'

Results window is blank.

A very simple smiley face, with SVG showing in the devtools

There's got to be a better way

Screenshot of the Potrace web page

Introducing SVGcode (svgco.de)

Screenshow of an image editing app in the browser. The image from before is being edited.

G'day, mate! Also available in en-AU and in dark mode!

the Window is upside down and in dark mode.

G'day, mate! Also available in en-GB and in dark mode!

The image is rightside up.

What SVGcode does differently

Offline-enabled PWA, runs entirely in the client. No server to trust.
Posterizes images and runs potrace per color, then reassembles the fragments for the resulting image.
Exposes all of potrace's options, including expert options.
Optimizes SVGs via svgo.

Installable, offline-enabled PWA

SVGcode is an installable Progressive Web App and therefore fully offline enabled.

The app is based on the Vanilla JS template for Vite.js and uses the popular Vite plugin PWA, which creates a service worker that uses Workbox.js under the hood.

Project Fugu

An effort to close gaps in the web's capabilities enabling new classes of applications to run on the web.

Cartoon image of a puffer fish.

Window Controls Overlay API

To maximize the available screen real estate, SVGcode uses Window Controls Overlay customization by moving its main menu up into the title bar area.

Screnshot shows before and after ise of overlay API.

const WINDOW_CONTROLS_OVERLAY = 'window-controls-overlay

const onGeometryChange = () => {
	const visible = navigator windowControlsOverlay.visible;
	menu.classList.toggle (WINDOW_CONTROLS_OVERLAY, visible);
	main.classList.toggle (WINDOW_CONTROLS_OVERLAY, visible);
};

navigator .windowControls0verlay. addEventListener (
	'geometrychange'
	debounce (async () => {
		onGeometryChange ( );
	}, 250),
);

Repeat of slide before code, showing effect of overlay API.

Repeat of code slide

File System Access API

SVGcode uses the File System Access API via the browser-fs-access ponyfill.

On supporting browsers, it can store a serialized FileSystemFileHandle to let the user continue where they left off after a reload.

This also works with the Drag and Drop API.

Screenshot of window with Dialog asking if you want to continue with summit22-yiying-header-square. jpg where you left off.

document.addEventListener('drop', async (event) => {
    event.preventDefault();

    const item = event.dataTransfer.items[0];

    if (item.kind === 'file') {
        inputImage.addEventListener('load', () => {
            URL.revokeObjectURL(blobURL);
        }, (once: true});

        const handle = await item.getsFileSystemHandle();

        if (handle.kind I = s 'file') return;

        const file = await handle.getFile();

        const blobURL = URL.createObjectURL(file);

        inputImage.src = blobURL;

        await set(FILE_HANDLE, handle);
    }
});

Async Clipboard API

SVGcode is also fully integrated with the operating system's clipboard via the Async Clipboard API.

You can paste images from the operating system's file explorer into the app either by clicking the Paste Image button or by pressing command/control + v on your keyboard.

Screenshot of open Mac file explorer window.

Async Clipboard API (continued)

The Async Clipboard API has recently gained the ability to deal with SVG images as well, so you can also copy an SVG image and paste it into another application for further processing.

copyButton addEventListener('click', async () => {
    let svg = svgOutput.innerHTML;
    showToast(i18n.t('optimizingSVG'), Infinity);
    svg = await optimizeSVG(svg);
    const textBlob = new Blob([svg], {
        type: 'text/plain'
    });
    const svgBlob = new Blob([svg], {
        type: 'image/svg+xml'
    })
    navigator.clipboard.write([
        new ClipboardItem({
            [svgBlob.type]: svgBlob,
            [textBlob.type]: textBlob,
        }),
    ]);
    showToast(i18n.t('copiedSVG'));
});

File Handling API

As an installed PWA, SVGcode can become a file handler, or even the default file handler, for image files.

This means image files can be opened in SVGcode right from the operating system's file explorer.

window.launchQueue.setConsumer(async (launchParams) => {
    if (!launchParams.files.length) return;
    for (const handle of launchParams.files) {
        const file = await handle.getFile();
        if (file.type.startsWith('image/')) {
            const blobURL = URL.createObjectURL(file);
            inputImage.addEventListener('load', () =>
                URL.revokeObjectURL(blobURL);
            }, {
                once: true
            });
        inputImage.src = blobURL;
        await set(FILE_HANDLE, handle);
        return;
    }
}
});

Web Share API

SVGcode can share SVGs via the Web Share API, for example, if you want to send the SVG to someone via email

shareSVGButton.addEventListener('click', async () => {
    let svg = svgOutput.innerHTML;
    svg = await optimizeSVG(svg);
    const suggestedFileName = getSuggestedFileName(await get(FILE_HANDLE)) || 'Untitled.svg';
    const file = new File([svg], suggestedFileName, (type: 'image/svg+xml'));
    const data = (
        files: [file],
    };

    if (navigator.canShare(data)) {
        try {
            await navigator.share(data);
        } catch (err) {
            console.error(err name, err.message);
        }
    }
});

Web Share Target API

In the opposite direction, SVGcode can also serve as a share target, so when you have an image file, you can share it to SVGcode.

{
	…
	
    "share_target": {
        "action": "https://svgco.de/share-target/"
        "method": "POST"
        "enctype": "multipart/form-data"
        "params": {
            "files": [{
                "name": "image",
                "accept": ["image/jpeg", "image/png", "image/webp", "image/gif"]
            }]
        }
    }

	…
	
}

self.addEventListener('fetch', (fetchEvent) => {
    if (
        fetchEvent.request.url.endsWith('/share-target/') && fetchEvent.request.method === 'POST'
    ) {
        return fetchEvent.respondWith(
        	(async () => {
				const formData = await fetchEvent.request.formata()
				const image = formData.get('image'):
					const keys = await caches.keys();
				const mediaCache = await caches.open(
					keys.filter((key) => key.startsWith('media'))[0], );
				await mediaCache.put('shared-image', new Response(image));
				return Response.redirect('./?share - target', 303);
	        })(),
    	);
	}
});