Micro FrontEnd Architecture at Scale

Ehsan Gazar: Does this technology work?

I think.

Yeah.

You can see the presentation.

Okay.

just a quick introduction of me.

I'm Gaz and we've got Shabnam here.

We're both technical leads at Mecca.

Who knows Mecca?

Okay.

It looks like, yeah, I can see.

Okay.

So Mecca is just an e commerce for all the makeup brands.

probably you've seen that they have more than a hundred stores in Australia and New Zealand.

So if you step into one of their stores, you know what they're about actually.

there's a website and we work mainly on the website and there's an engineering team.

So we're going to talk about the approach we took for the frontend of Mecca, which is, we use the kind of a new approach, it's called Microfrontend, right?

So this is going to be all the talk all about.

and then what's going to go for the 30 minutes.

So we're going to cover all the topics here, right?

30 minutes is probably not enough to cover all the challenges we faced through the journey, but probably we can wrap up with the keynotes on some of the good points, and hopefully if you have any questions after, we can answer those as well.

first I'm going to talk about, what is microfrontend, just for, those who don't know what is, what it is.

Who knows, who heard of the microfrontend term?

That's, good.

And who is actually using microfrontend?

I think there are some, okay, that's good.

There are some, questions about, okay, what microfrontend means, and it can be like different definitions to it as well.

So we're gonna go through that as well.

then I'm gonna pass it to Shabnam, talking, about the challenges we face through and the solutions we came up with.

And, she's also going to take us, to the journey of like how we actually did it and what's going to be the future of how we, what we're planning to do as well.

And then I'm going to come and wrap up the, whole architecture and give, you guys the pros and cons of this solution versus the, old way.

what is Microfrontend?

Let's imagine, as always, we have a website, and the website is just one source of code, and this source of code, it just is in one repo.

the thing about it, it's just one single repo, you might have it in Bitbucket or GitHub.

Which is easy, as always, that's what we're familiar with.

the next thing is the routing.

what we're doing in the routing, if you're using, for example, Next.

js, we're using, our own routing, or we're using, Remix these days, Whatever we use, we are also handling routing as part of probably frontend.

You might go a little bit extra with NgiEnix setups or some of the other stuff, but most of the routing is actually in frontend in that repo.

The other thing is the UI components, which...

We love to have the site, site system and go through having the components and all the library and we usually put those in the same repo as well.

The other thing, if you have a logic, if you're, it's a gambling website, it's an e commerce or it's any sort of other websites, you have a logic and logic goes to this folder which is also sitting in, for example, frontend usually.

Hopefully you have enough unit testing coverage there as well.

and.

If you fancy, you go with Redux or XState and anything else for handling state management.

But the key is, everything is in one repo and it's simple because this, everything is one place.

We just probably have different folder structures, some people might prefer something else, but this is what we're going to have in a monolithic approach.

But what we're going to talk about here is a micro front end, which is, basically starts with the definition of having different apps, different repositories, different sections of the website.

So it depends how much you want to define it.

You can define it as, big as, oh, this is a separate, complete website.

Could be micro frontend, but they're not really micro.

And it could be as small as pages, say, oh, homepages, one app, and other ones, for example, cart, bag, checkout, they can be all different apps as well.

And then they all talk to the backend, so it's a completely different approach.

It looks like we have a simple HTML, but this time we have actually five HTML files.

If you look into, for example, ten years ago.

And this is the micro frontend.

Hopefully it gives you enough to do the rest of the topic.

And I'm going to pass it to Shabnam to talk about all the challenges and solutions she came up with.

Shabnam Mohammadian: thanks guys.

our story start, about two years ago when we wanted to, migrate a monolith app to a micro frontend structure.

so the first step that, we approached was, how to structure our micro front.

And so there is a common approach that you would gather all of the, shared concern in one place.

For example, you are going to have header navigation as one app or package, or you are going to have search as another one or cart and so on as another, micro app.

And then you need a container app or shell app to integrate and orchestrate all of these applications.

But, before deciding on, what's the structure of, the, your micro frontend is, we need to answer two concerns, two main concerns, because whatever structure you are gonna take is gonna dictate your next challenges along the way.

the first thing is about how to integrate these micro applications together.

there are two solutions, whether you want to do it at build time or run time.

If you want to do it at build time, it means any changes to any of these apps, it means, rebuild of all of these applications together.

It means lots of cost and time consumption.

The second way is to, integrate them at runtime.

Again, if you want to do it at runtime, there are two approaches.

One could be at client side, the other one could be at server side.

If you want to do it at client side, easy, there are lots of ways to do You can use iframe, Web Components, Simple React app, or module federation.

But we are an e commerce, that's a no go for us, because SEO matters a lot for us.

So we need to go with server side way.

Two ago, when we wanted to start this project, Module Federation has introduced a server side render of its version, which was at its earliest stages.

And really, it wasn't a production ready, of course, for an e commerce website.

There is another way to, write a streaming server to do all of these integrations at server side during runtime, which is going to add lots of other complexity, to all of the things we have in microfrontend.

The next question that we need to answer is how to, how these applications should communicate with each other.

In other words, how to keep a reactive state to, handle all of the interaction within these apps to get whatever has changed in one app in another app.

So what a common app, common approach is to use custom events.

So you are going to end up with, lots of event listener and dispatching events across the microfrontend.

The other way is to use, props and callbacks.

No one would like to end in a hell of props, really.

There are other ways like WebWorker, PubSub library, which you can use.

But the approach that we have taken at Mecca, it's basically each page, it's, is its own application.

We have one app for homepage, one app for, product detail page, another one for checkouts and so on.

And they're all, sharing a pool of, dependencies.

In this case, header, search cards, all of these are going to be a package.

we have been versioning all of these packages.

I got this question a lot during these two years.

What, why do we need to version these packages?

Versioning packages has helped us in two ways.

In one way, we could achieve consistency throughout, our micro front end.

So simply, to pushing changes and automatically through our pipeline is going to integrate and update all of our packages.

and by major changes and breaking changes, we can, gradually in isolation test our, features and release them independently, independently, like, an agile way.

this structure has really simplified our state management.

So we have a state provider is wrapping up, our application.

And as the user navigate through this micro application throughout our website, our state management is going to be persisted through session storage.

So it was really easy for us to handle.

The other challenge in micro front end world is to keep consistency.

So it's all about customer.

Whenever a customer comes to your website, they shouldn't feel like, the website is coming from different places.

We need to keep the website consistent.

in order to achieve that, we are using a shared UI library with a shared theme provider, which is feeding, our design system to our application.

while we are at consistency topic, the other thing is to, keep the apps and packages configuration consistent as well.

So we have written, multiple scripts.

So easily by running a command, you can choose whether you want to create or, an application or package and easily in a couple of seconds, it's going to create the app or package for you.

The next step is routing.

in the common structure that I showed you, there is a container app or shell app, which you can centralize all of your routing logic there.

But in our structure, we don't have any, container app or shell app.

And we have a quite, complex routing logic.

For one thing, we need to have shortest path as possible because of SEO.

And also we are handling internationalization throughout our routing.

So we ended up using ReverseProxy, all of our, all of the requests from browser are going to the ReverseProxy, and from there, it's gonna, from the, all of the logic stack, it's gonna decide which app, which container should respond to this request.

And for this ReverseProxy, we are using a lambda at edge, which is giving us less latency by running it at edge close to user location.

one of the other benefits of, Microfrontend is to achieve higher scalability and fault tolerance.

if we have been in the old common structure, if you have, for example, a search application or cart application, whenever you want to, scale up one of your pages, like homepage or product detail page, you need to scale up everything.

Kind of like a monolith, mindset again.

But for us, it's easier, for example, throughout certain campaigns, at certain period of time, we need to scale up some of our pages.

So it's easily done in this structure, it has saved us a lot.

And if one of the containers is not healthy, it's only going to bring down one page rather than bringing down the whole website altogether.

whatever we have discussed so far, it was just the tip of the iceberg.

We have achieved autonomous team working on, independent feature, releasing them gradually.

But one of the main challenges for us after we started this project was the development and deployment complexity.

So whether to choose going with monorepo or polyrepo.

But before answering this question and how we have resolved this, we need to think about this question, how to share code between these, micro applications, how to prevent duplication of code, and how to symlink multiple packages and apps during dev time.

I know there are lots of packages doing the job like ??? or some other stuff, but still they are having issue with operating systems, some of the operating system, or if you want to release multiple application at the same time, how to organize this deployment, if they are sitting in different, repos, and also how to keep these apps and packages consistent.

for example, last week we have update into node version 18, and, if you are in, polyrepo, we had to do it one by one in each of our repo, and there is a chance that we missed one and we lose the consistency in everything.

so in Mecca, we have chosen to go with monorepo.

We have started in 2021 with Lerna version 4.

it has given us lots of benefits and lots of, challenges.

So we have a single repo at the moment with 16 applications, 89 packages, quite a big monorepo, about more than 20 front end developers working on the same repo.

we were able to manage, sharing code between these micro front end.

We don't have any code duplication, anything that has been, used twice in any of two apps, it has been captured in one package.

And Yarn Workspace and symlinking is working really nice at dev time, we can easily watch the changes with packages and apps at the same time during dev time.

versioning of our packages that I earlier mentioned how we are using it.

It was really, it's really easy, it's easily through through the conventional commit, it would understand whether it's a minor change or a major change and throughout, our centralized CICD pipeline, we can easily, identify minor changes and, rebuild and deploy all of the consumer apps.

and, But, going to the, some of the challenges that we had, here.

So when we started, we had really brutal code space with unclear boundaries.

We have resolved this by defining better our squad and our team.

Broken master, it means everything is broken and you cannot release anything.

we have improved our CI CD pipeline with better testing, linting, everything, and checking our builds to make sure nothing broken is merged to master.

we have, long CI time because even with a small change, all of the consumer were getting updated and deployed.

We had resolved this, problem.

I'm going to get to it.

And.

the next one, which was really challenging, has taken us lots of time is how Node.

js is resolving the, it's node modules in a flattened dependency tree, which to be honest, it doesn't work with well with monorepo and we got lots of random error of non found, dependencies.

So Basically, one of the benefits of what we wanted to achieve here with Microfrontend was to get the flexibility of TechaStack [???]. So to use other stack or other frameworks in the other applications.

But with this issue, we have been locked here, but we have a solution for it.

in the beginning, developer experience was really bad, because it's a quite big repo, you imagine whenever you pull master, it's quite big.

And All developers need to build this monorepo in their local and, in their local machine and start doing that, which we have resolved with Enix.

We've been really lucky.

We have been facing all of these challenges, Enix took the ownership of Lerna and we are using the, benefit of distributed caching of Enix Cloud, which has saved us a lot of time in our CI pipeline and at dev time.

So if, for example, I push something, I built something, test it and push it to, push it at the repo.

And next time, Gaz, pull it down.

He doesn't need to build it again, he's just gonna pull the, build from Enix Cloud.

And if there are some independent tasks, like unit tests, build, or anything that are completely independent, they can run, in parallel.

And yeah, it has helped us a lot.

We are not a sponsor by Enix, but, I don't know where we have been gone without Enix now.

It's such a big monorepo.

the next thing for us...

What we have done so far is adding PNPM or replacing YARN with PNPM in our monorepo.

We have done some investigation.

One of the main bottlenecks of our CI pipeline is still installing, is the installation of node modules.

We have tested this, it's going to be three times faster with PNPM and it's going to save some disk size because, disk size, because, it's hard linking all of the dependency from PNPM store and if you have already installed something on your local machine, it's not going to install it again.

It's just going to link it from there.

If you can remember, one of the thing that was the main issue for our monorepo was how NodeJS is resolving the dependency, which PNPM has resolved this by doing same linking of packages.

So we are hoping that we get back our freedom by replacing Yarn PNPM.

one last thing.

The, process of improving monorepo and the performance of Monorepo is an ongoing one, and we are just in the middle of it.

We're two.

Ehsan Gazar: Can you hear me?

so first I should say thank you.

Thank you for Shabnam.

Thank you.

For, talking about all the challenges.

the challenges part I wanted someone else to talk about because it was actually hard to explain.

It was, thank you.

so what I'm going to talk about is just, put these things together as a summary.

the team behind it is, I don't know, we've got many engineers working on these things.

and then each one of the teams has got responsibility for each one of the apps as well.

it's a massive effort by the team.

And they're doing great.

I'm gonna talk about, generally what happens in Mecca now.

So Mecca is a website, and that's our, that's our red logo.

And if you go after, if you want to visit the website, first thing you go to CloudFront.

So basically for caching, all the rules, it goes there, and yeah, obviously Microfrontend caching is a bit of a challenge as well, which we didn't really go into, but yeah, that's a challenge.

and then it goes to the Lambda, which we talked about.

It's a Lambda edge, so we can get a better speed through different locations.

And the Lambda, it's responsible for the routing and it gives you direction to which app you want to go to, because we can't have it in the app itself.

And after that, for the scaling part, it goes to this load balancer, which...

If all of you decide on a Boxing Day to buy everything from Mecca, probably the pay the checkout app needs to be scaled.

And thank you for buying first and, we have to cover it.

So then it goes to all the apps.

We have many apps and each one of the apps are responsible to do part of the website and they're NextJS but they're gonna be deployed in ECS.

the pipeline and DevOps team are doing an amazing job over there, actually.

Because building all of these apps and then modules behind them and everything is actually quite hard as well.

Then the request goes to the app.

When we decide to go which app, we decide it in the Lambda.

So now we are in the app and each one of the apps has got all the npm packages behind it.

Which we have so many now.

And then each one of them are being versioned one after another.

If you have a tiny change or if you have a breaking change.

Obviously, if you have a breaking change we have to upgrade the apps slower which gives us better, better and more stable app.

But, managing all of these Node modules, which we call them, packages, it's a bit hard as well.

So that's another one.

And after all of those goes to our, backend for frontend layer, which obviously has so many services in the core and we don't want to request them, which is for security reasons, also mapping the requests and everything.

So we also have the BFF layer and there's a team that they're working on that as well.

And after that, it goes to our service layer, which there are many more apps and many more marketing services and stuff.

Thank you, Linda.

Linda is pointing at five minutes, so I should wrap it up.

Thank you.

and then I'm gonna quickly go to what we learned through the journey, and probably it's also a comparison between what is, microfrontend versus the monolithic approach, right?

we've got, we can have smaller repos.

We went through a monolithic approach, but if we wanted to, we could have, smaller repos.

Which is good, if you think about each one of the apps, it's just if you run them, it's just one or two pages.

And, they're just using all the NPM modules behind them, which is awesome.

it gives us benefits.

The other one is, if something breaks, probably we are fine.

she was looking at me a few times because, obviously, I was the responsible one for breaking the code.

but if we break the code, we're fine, right?

Hopefully, just one app breaks and not the other one.

so it's better.

It gives us better, stable versions and it's awesome.

And the scalability is, great as well.

I think DevOps love this part as well because...

Just one app is gonna be a scale, and they're not thinking about pricing of the bigger, massive apps and memory usage and everything.

It's just a scale, simple one, and just two, three, and the load balancer handles everything.

the work distribution between teams is way easier because they know their part.

We just give them, oh, okay, you're working on homepage and checkout, and you guys are working on product.

I'm stuck here, actually.

Okay.

and then superior and explicit builds.

The builds are much faster because we don't have to build a whole app...

I mean, if you're using...

If you're changing something, a package, that probably is gonna, change all the apps, then you're gonna have a problem because the build's gonna be probably within an hour or so.

But then if you're just changing one app or just one module, you don't really have to build everything.

So it's way faster for the builds and we can actually use these features.

But...

I imagine not all of us are working on crazy projects with many engineers, so we don't really have to go to, through the whole complexity.

So it's not really ideal for the small projects.

I don't really recommend it.

the consistency through the apps can be difficult because imagine you have a massive version for one of the packages and you don't want to upgrade everything with it.

And one of the apps is using the new one, not the other ones.

And you have to remember to upgrade everything.

there's less consistency and, CI CD is absolutely hard to implement.

Because one way was just deploy everything to Vercel and no, not worry about anything.

But now you have to implement the whole thing.

and...

A start is actually, expensive, because it feels like you're building everything, and the build time, and everything, and CI, and DevOps, and engineers, the complexity, and, yeah, it's also, it's a bit of a pricing problem there as well.

I'm gonna leave you guys with this question.

I think the complexity of frontend is just, getting, worse and worse every day.

And one of the questions now we have is, are we going to the...

monolithic approach or micro front end.

And if we go, do we go monoreport or do we go polyrepo, which is also a question every time if we want to start a project.

thanks again.

This is, the credit goes to all the team, which some of them are actually here.

Thank you.

And Shabnam and myself, you can reach out to us in LinkedIn and we're gonna be around here as well.

I think we have everything.

yeah.

And thank you.

APPLAUSE