The State of ES Modules

Top five, issues in all our JavaScript projects, ES modules.

I hope you're not suffering from vertigo or anything like that, because this is going to be quite a journey.

That's why I chose this video.

so I'm, Software engineer, working at Seek.

We, work on, tooling and few open source projects you might have heard of, Braid or Playroom that, Mark has, has mentioned before.

And, as part of that, we went on a bit of a journey and, this is me sharing the story of that journey.

So we'll talk about why do we care about ES modules, the differences between ES modules and CommonJS modules, if you're familiar with either, the situation in browsers, bundlers, Nodejs, TypeScript, and then what it takes to actually support ES modules.

What are ES modules?

it's basically this.

The way of importing stuff from one file and exporting something from one file and importing another file.

Basically, it's just two keywords, import and export, you just put that there, magically works.

Similar to CommonJS, it's module dot export something, or require something.

Simple, simply like that.

The difference is, there are many differences, but Import and export can only be used at the top level.

But let's see what, let's dig a little bit deeper into the differences.

ES modules can be synchronous or asynchronous.

And we'll see what that is about.

ES modules can be used in browsers, and CommonJS modules cannot be, because you need something to hook up those, exports and, require calls.

ES modules can import CommonJS modules, but not the other way around.

So ES modules can import CommonJS as well as ES, other ES modules.

those keywords import and export only at the top level.

There's no global, those magic variables, but you can create them, with, from import dot meta dot url.

there's no global require, but, again, you can create one.

And, very importantly, there's this thing called import specifiers.

Which is the thing that you're importing from, and those must have a file extension, but we'll get into detail in the next slide.

by comparison, CommonJS is always synchronous, which makes them slow, this also makes them hard to statically analyze.

Hard, but not impossible, because you can have require anywhere, it can be conditionally loaded, stuff like that, so it's harder.

you can't, use them, in a browser, without, without the build step, you can't require ES modules, we can, you can, import them, and that's done asynchronously, and then, with require, you can omit the dot js extension.

let's see, what we mean by eSM can be async.

there are three steps, loading, the files from the file system and executing them in the browser.

first you need to find them, find all the files, download them and parse them.

then you have to reserve memory in, in, for them, load those little boxes there.

So that's where, that's where the values will be, but that's, in a separate step, which is after evaluation, you have the values and you link everything together.

this is taken from, from, hax.mozilla.Org blog.

it's quite, quite an interesting, deep dive into ES modules there.

and let's talk a bit about import specifiers.

As I said, it's the thing that you import from, so you can have relative ones, relative import specifiers, which starts with a dot slash or dot slash, absolute, which is a slash, then you have protocols file or HTTP or something.

but then the problematic ones are the bare module specifiers, which is none of the above.

you, when you do import from React, that doesn't, you have to do something to make it work in the browser.

It needs to, there's a bit more meaning there.

yeah, there's bare, and then there's a subpath, which is slash something.

Then you can also have a scope, but that's node specific.

So those, are all, bare module specifiers.

why do we, why do we care about ES modules for all of this?

because import and export can only be used at the top level, that makes them statically analyzable, which, makes it faster to traverse the module graph.

That also enables three shaking and which makes the fo code faster to execute as well.

And that means you can also do synchronous or asynchronous because you can split those steps, and you can run them asynchronously, which means that you can use them in the browser.

'cause you don't have a, you can't wait for the thing to, to execute in the browser.

So you have to do it asynchronously, which also means you might not need a bundler, which is, which is exciting, right?

Let's, let's have a look at, ESM in browsers.

this is how you would use ESM in, a browser.

You have your script type module, you can import it from an external source, or, you can write code in line, you can use the same syntax, and you have a relative import there.

Easy, right?

Things get complicated, as I said, when you use bare import specifiers.

when you write such code in, a browser, the browser doesn't know what React DOM is.

you need to tell it, where do I load React DOM from?

What is that?

and that, there's a bit of foreshadowing here, but there are two options.

One is, you could rewrite your, import specifiers.

So the code above becomes the, code, below.

you just rewrite it, you tell it this far, this thing, you, basically, you need to have a, step that evaluates that and changes it to, an actual path through that.

You say you, if you upload your node modules to S3, then you can just say that's where it is.

and that's what Vite does, and, Vite is a new, new kid on the block, that's what it does in development, it just rewrites your, import specifiers so that they actually serve from Node modules.

and the other option is import maps.

basically you tell the browser what each of those, import specifiers are, which, where they're mapped.

And, then it works, it's widely supported, it's, it's a standard and it also works in, in, and the, the browser support is actually pretty good, Safari was the last, excuse me, the last holdout, March, April, something like that, this year is when they, supported, started supporting import maps, you can use them today, so it's, pretty good if you don't have to worry about IE 11 or anything like that.

There's also a few options, there's some CDNs that do this for you, so you don't have to, traverse your module graph, it, I'm not gonna spend too much time, talking about, ah, there's, the CDNs as well, as you can build your own, tool, you can just download NPM packages and rewrite the import specifiers and you can, go that way.

but, I'm not going to talk too much about it, because, after this, Omar's going to tell us all about import maps, so stick around for that.

let's see what the situation is in Node.js, because, this one is, quite a journey.

how many of you guys know this person?

A handful.

you should, you should know who this person is because he's in all your projects.

he's that guy over there in the little, the little brick over there.

as you can see, he's very prolific.

He's, nice guy.

He caught a little bit of flack.

We'll see why.

Caught a little flack for, some of the moves to ESM.

I don't know if he still is doing JavaScript development.

I think he's moved to, to, Mac, but anyway, he's got a lot of packages and, one day he decided he wanted to, move to ESM because it's pretty cool, it's, better and he wanted to migrate his packages to ESM.

So he was like dying, right?

This was January 2020, 2021.

And someone says, yeah, go for it.

This is a great idea.

so he's yeah, I'm going to do it.

This, and he wrote, up his thoughts.

there's a discussion.

it's still up there.

I think there's thousands of comments.

I don't know.

There's a lot of comments there.

but, he explained what he was going to do.

He wrote up a, a Medium post that, he wrote his thoughts there and, essentially there are two ways to go about moving to ESM.

You can go pure ESM, you just switch to type module and, Do minimal changes to your code, or you can do a dual ESM CJS strategy, which is, has all kinds of downsides.

He wrote about it, about, about them there.

and he essentially said, nah, I'm going to do option one.

rip off the band aid and move the ecosystem forward.

And, it was not, it was not well received by, by the community.

let's see.

I'll handpick a few.

this, one mid last year, August, 2022, don't, please don't do this.

This is not, this guy is on the technical steering committee for Node.

So he's don't, please don't do this.

This is not, it's not a good idea.

Pete Hunt, who you might know from React, he was super frustrated.

He's he started forking some of those packages and republished them under the scope, actually works.

Yeah.

Devon, the creator of Parcel, maintainer of a few, open source projects as well.

Again, struggling.

This is December, 2022.

struggling with ESM support.

They went with, he went with, dual ESM CJS.

he cracked it later, what was this, February 2023, so recently.

These are highly capable people, not like randoms, people that are struggling with the state.

stuff like that, like this is how much.

And this is not even close to, how all of this is needed.

This is the, this is how you, where you start from.

This is the minimum.

Mark Erickson, the, Redux maintainer, again, struggling to do all of these things.

Recently, April 2023, ESM, CJS, all of this matrix together.

It's a miracle anything about this ecosystem works at all, right?

Frustrating to deal with, this is getting utterly ridiculous.

ah, Mark saw, an advanced screening of this talk and he had, a thought about the ES modules as well.

let's see.

What it takes to support ES modules.

so as I, we start from there, this is, the, basically the minimum, but this is showing a dual CJS ESM strategy, because, we'll get to, towards the end why, you, unless you want to be like Sinder and, get a lot of flack, you probably want to do this, because you don't know who's going to consume your package.

if you do, then you can just go pure ESM, but let's start with, this as a baseline.

You have your main, we know about this, it's the, old way, if you have to support node versions lower than 12, 12, 20, 20 or something, you have to have a main, newer version, oh, there's module for bundlers, we'll talk about bundlers in, a minute, TypeScript, and we'll talk about TypeScript as well, then you have exports, Newer versions of Node support exports, so there's overlap between exports and main, so if, you don't worry, don't have to worry about the older versions, then you can just go exports.

However, tooling is lacking some support, so you might still want to have both main and, exports.

And then in exports, you have to expose your package.json because that's the way it has to happen in ESM.

If you don't have an exports, then you, someone can't load your package.

json.

So tools will have to, some of them need to look at your package.

json to read stuff from, there.

If you don't expose it, then they can't.

And then, say you have multiple entry points, like you, you don't just have the main.

You have demo slash other, then you have to have TypeScript, you have to have a version for ESM, you have to have a version for, Require for CJS, and then for, again, for all the versions of tools that don't understand, exports, you have to have a fallback, so you create a folder with packages in there, and then you just redirect to where your actual file is.

This is, again support importing from demo/other So, ignoring the JSX transform, let's say you wrote some, TypeScript, you probably wrote something like this, right?

You have import star as react from react, so you have React and then React.

on /server, what would you, if you were a compiler, what would you compile this to?

you would probably, this is the, important part, we don't worry about the, JSX.

You would probably write something like this, ignoring, again, the, JSX transform, you would probably transform it to, the same, it's just mjs, the extension mjs because it, it's ES module.

that doesn't actually work, in React 17.

In React 17, you have to point to an actual file, because there's no exports in React 17.

So the React 17 package does not have exports, and there's a, there's an issue about this, probably hundreds of comments there.

it's a breaking, considered a breaking change, they're not going to do it.

So if you're supporting, if you're building a design system that, If you want to support React 17 and 18, tough, it's either or, either that or the other one, or you can solve it, at bundle time, but, essentially, yeah, that's, that's the situation.

You can't, you can't have it both ways.

so speaking of bundlers, let's see what the situation is in bundlers.

good news is it's widely and well supported because before it was supported in browsers and as soon as the standard was, the spec was standardized, bundlers went in and supported it and they invented this faux ESM thing and the module entry, the module field analogous to main.

Babel and Webpack supported it.

And You might have noticed there was no, Parcel, that's because Parcel very recently supported, exports in it, so it's an important part of ES modules, right?

there is an analysis by one of the, guys that, works on TypeScript, and I've got a, got a little recording here.

the maintainer of, the creator and maintainer of Webpack, wrote up this interop test and I've got a recording to see how extensive, how much, how many differences there are.

So there's there's a matrix of Babel, ESBuild, Rollup, Node, Webpack and all that.

There's a lot of stuff, there and there's a lot of, it's mostly green, like it's, pretty good.

But there's still, a lot of things to worry about if you're running, writing a bundler.

The one particular bundler I want to call out is Vite, which we started using at Seek.

We've started upgrading our tooling and we're looking more at Rollup and Vite and ESBuild.

And it's, it's pretty good.

So it supports ESM out of the box.

And, as I said, it pre bundles, updates your, Import specifiers, it changes them so that it loads from the file system.

So it's, ESM first.

And there's a few projects that, use it under the hood.

So based on it, Vitest is one of them.

it's Vite Team's, answer to Jest.

They're trying to build something that supports ES modules because ES modules in Jest is a problem.

We'll see, in a few slides.

And then there's Vite Node, where you can actually use Vite as a runtime, which is crazy.

Mark has done some, some experimental stuff in, Vanilla.

You can, you can ask him about that.

And, ThoughtWorks thinks it's pretty good.

they recently promoted it to, trial.

So check it out.

Yeah, Vite is, it's pretty cool.

All right.

Browsers, great.

Bundlers, cool.

Node, not so much.

TypeScript, let's see what the situation is in TypeScript.

there, these are the important ingredients to use ESM in TypeScript.

You, you gotta have a type module in package.

json, this is, this has to happen, this is how they want it.

then, which has also this, the side effect of dot js files are treated as ES modules, You have to think about that.

it's not just TypeScript.

so if you want to go CJS or or ESM profile, you can do that with the file extension.

then you can use imports and exports, defined and package json.

there's a, there are a few, slight differences with import specifiers and then there's new extensions like MTS and CTS, which are the, equivalent of.

mjs and cjs.

And then there's also do d mts and do cts, which is for declaration files.

'cause we want more and more files and more extensions and, we can have a look at, a few, TSS config options to make our lives, a bit easier.

if you wanna use ES modules, what does your TS config look like?

probably something like this.

This is the, minimum.

You have to have module node 16 or node next, even though we're on node 18 now, that's how it is, node 16.

Then module resolution, again, node next or node 16, or there's bundler, which is a very recent addition.

going back to our package json, we, multiply the, each of the, conditions on every entry by two, because why not?

And so we have type module, as I said.

we're just going to talk about the export, conditions here.

So we've, we have the new extensions.

We want to do the dual, package, dual ESM CJS strategy.

We got to branch off and do types for the ESM version and then types for the CJS version, something like this, It's obviously the same for every entry point that you want, so you have to multiply that and you have longer package jsons, and then again, for versions of TypeScript, because you don't know where your code is going to be consumed, you're going to write TypeScript that might be used in an older version of, in a package that has an older version of TypeScript.

You don't know.

so you have to provide this fallback again.

And then, older version of TypeScript don't know about these new extensions.

So you might have to provide, the, same contents with, for, with the old extension.

So it's, it's, not great.

So import specifiers.

Let's say you have foo dot ts on the disk, and then you're, you have this sort of code in bar.

ts.

Which one is valid?

What do you think?

not all of them, just the first three, but actually all of them can be valid.

If you have foo.

mts, you can import from mjs, you can import from foo.

ts with the extension, without the extension, and with js, which I found a bit surprising when they first announced it.

with the simplest, Module, node 16.

This is what, this is the most basic setup.

This is what your code would look like.

Notice there it's foo dot js, so the file is food dot ts, but you're importing from foo dot js.

Does that make sense?

Not to me, but to the TypeScript team.

It does.

And they thought, Write code.

We don't want to touch your import specifiers.

You're not going to change anything about them.

Write the code so that it is valid after you compile, because you're going to compile the code, right?

So it's foo.

js is going to trans it's going to foo.

ts is going to be compiled to foo.

js.

So your code is going to eventually be valid.

But if you have ESLint rules, then that's that doesn't exist.

That doesn't make sense.

yeah, that's the situation, 4.

0, TypeScript 4.

7 introduced, this.

these options, this option, this is the kind of code without any other changes.

if you do want to have extensions, but you want to have the real extension, there's an option for that, there's Allow Importing TS Extensions, and this is what your code would look like.

personally, I'm not a big fan of that, but, if you like seeing extensions there, that actually points to an actual file.

With Module Resolution Bundler, You probably write such code today, because TypeScript allows you to write such code.

You don't have to change anything, so just throw a module resolution bundler in there, you can keep your code as is, just upgrade to TypeScript 5, and everything is fine.

Thank you Microsoft, because they listened, because in 4.

7 there was the js extension.

Sorry, everybody was a bit, what is this?

And then in TypeScript through five, they actually listened to the community.

So that was good.

alright.

What else?

What else is there in TypeScript?

you know how we have that, crazy duplication of files.

You have dot mts, dot do dot mts, . there's both that, for backwards compatibility, but, so yeah, there's, the types for that.

You have, dcts, dts, and again, with the fallback, what if I don't want to do that?

there's an option for that, resolvePackageJsonExports.

You have that in your, in ts config, and then you can get rid of a lot of the stuff.

But again, depending on who's consuming your package, then you can, you don't have, if you don't have to worry about backwards compatibility, then it's all sweet.

Simplify, it simplifies a lot.

I guess the TLDR is, go upgrade to TypeScript 5 because it's, it's pretty good.

wrapping up, are we there yet with, after everything?

there are still some issues.

a big one, which probably a lot of people are using, is Jest.

Jest does not support ES modules.

You have to do something, transform them, because of, there's some, issues with Node that they depend on, and the code coverage as well, so they can't really do much about it without the Node, implementing those things.

so that was 2020, when those issues are still open, right?

There's still comments, there's, I don't know, hundreds of comments there.

the move to ESM, January 2021.

Recently, still issues popping up, I just, this was one that I picked, still popping up, probably every week, I don't know, depending on how big projects are, there's always something breaking, but there are also issues being resolved.

it's not just negative.

There's also, we're getting closer and closer like Parcel, as I said, recently, they've had, they've, it was a big, release, they wrote a new, resolver for, to support this, the export, we're getting, closer and closer.

It's, pretty good.

How close?

1 in 10 packages.

1 in 10 popular packages, and there's a definition for popular.

1 in 10 is ESM only, and then you have, 6.

7, dual, and then you have the 4.

As you can see, it's slowly growing.

This is every, 3 months or so, 4 months.

we're at around 10%.

We're, we're getting there, but we're not quite there yet.

That's it from me.

I hope you don't feel like you're trapped in an Escher painting.

it is, it's quite gnarly out there, but it's getting better and it's, we're getting closer to the ideal.

Thank you.