Scaling web development with a Micro Frontend Architecture

10_Grant Sheppard: Hey everyone.

Thanks for being here today.

Disclaimer, this is also my first conference talk.

So Henri did mention that he might spill some water over my laptop or saying this is a gentle introduction, but everything seems to be working so So I thought I'd just launch into a little bit about REA, just to get the ball rolling.

Most people when I explain that I work for REA group, particularly if they're outside of tech, they just sort of blank look on their face.

They don't really get what that is.

And then I mention that we run the realestate.com.au website, which most people understand.

As Chris pointed out, we were the 15th most trafficked site in Australia for October.

If you ignore all the international sites, it would've been the fourth.

If it wasn't for that terrible weather we've been having, we would've been third.

'Cause BOM is like right up there.

So just unfortunate timing really.

And just a little bit around I guess the, growth journey that we've been through.

Like all good responsible startups it all began in a garage with a couple of guys back in 1995.

And that sort of quickly grew to we're expanding pretty quickly.

We hit the two thousands and into 2010s and we're going through all those things that all the companies we're going through.

So we went all in on Agile no surprises.

We embraced microservices.

Which is pretty topical for this talk almost.

We adopted the DevOps, "you build it, you run it" approach.

And we absolutely went all in on the cloud and just put everything up there that we could.

And then by around 2017 we'd grown pretty substantially.

We're up around ballpark, about 42 squads that were delivering products.

We had about 500 plus systems.

I know that just because we do have an internal systems catalog where we track all of the systems that power our, services and products, and we're spread out across about a hundred or so different AWS accounts.

And then, that was all expected really because we were very much around enabling teams to be autonomous.

We were, that was expected and that was one of the things that enabled them to build their products fast and deliver really quickly.

So that explosion wasn't really unexpected.

By 2019, we'd had a bit of a change in our tech strategy and we'd managed to reduce that AWS explosion down to a more manageable level.

But the number of systems had just gone crazy.

I dunno what the exact number is by then, but it probably would've been well over a thousand systems that was predominantly powering our, main website amongst a few other things.

So really where we found ourselves was that as we're scaling, it certainly wasn't all, unicorns and rainbows.

We were very much there on the left where teams were spending time solving customer problems, but there was a bunch of other stuff that they were doing.

So things like choosing what framework to, to run their their Web app and what libraries how were we gonna deploy this thing?

What CI pipelines were we gonna use?

How were we gonna test it?

They're all important things, but they don't really add much in terms of the customer value.

And then on the flip side, we had people that were jumping into large sort of code bases owned by other teams and having to learn how those code bases worked and work within those processes.

So that's where we were, but we very much wanted to, or would prefer to have been over on the right hand side where we flip that around to spend more time solving customer problems.

So I guess just as a sort of over the past five or so years, we've done a few things to try and help with that.

We introduced a decision making framework for the idea being there that if we had more transparency over our decisions, it would really encourage better decision making and hopefully stop people making decisions or deciding outcomes that had already been decided before.

We started measuring our system health.

So all of our systems go through a quarterly health check where teams have a series of criteria where they have to rate their systems and that gets fed back into a central system.

We can then pull out a whole lot of insights out of those outta that data and work out where do we need to spend the time to improve the health of our systems.

We introduced this thing called 'federated delivery', which is about as fun as it sounds that was there to help teams that were, again, having to go into other people's code bases and if they had products that lived in those code bases, how do they go in there and make changes and work within the bounds of all these other teams all in there at the same time?

And it's, pretty messy.

But we, tried to put a process around it.

And we built a internal tech radar, so just like the ThoughtWorks tech radar, but this was internal to REA, so it was specific to the technologies that we use.

Again, the idea was to help guide people in, choosing what they should be using.

And the last thing we did was we, built this internal brand called 'CoLab'.

And CoLab is really just a, you can think of it as a, suite of internal products to help do a lot of those things that that you don't wanna have to have teams reinventing.

So whether it's how you deploy your app into the cloud having a design system for, your Web apps, it's all those products that can be used all across the company.

They all fell under that banner of CoLab.

And these things did all help.

We had, we managed to shift the dial.

Teams were spending a little bit more time solving customer problems, but the other stuff, it was still quite a fair chunk of their time.

So the textsprawl was absolutely still there.

We had all these things in place, but teams were still having to make a whole lot of choices.

Whe whether they were spinning up a new Web app, or whether they're going in and wanting to add functionality to another Web app, there was still a whole lot of decision making that had to be made first before you could really get started.

And the repetitive plumbing.

The things that don't really add a lot of value but are, necessary.

Like for example here, if we need to send some data to our analytics server, which we expect all of our Web apps to do, so we can see how, they're being used.

If you had three Web apps, you almost guaranteed to find three different implementations of, that integration with that server.

And although we'd introduced the design system, the UX consistency was better, but not there.

So this is just showing the what is it, six or seven different headers that are across our website.

So I think for a lot of people, they look realestate.com.au and they think, oh, it's one system.

But no, there is a lot of different Web apps behind the scenes.

And if someone wants to, a product person wants to update their header with something new and fancy, there was always decisions around how many systems will we roll out across into?

And you just end up with a really disjointed experience where there's, you jump around pages and the header and foot to keep changing and it's not particularly nice.

So none of this was that surprising really, because we had very much encouraged teams to, have that high level of autonomy.

Go in, build your own apps, in your own repos.

You own them entirely.

You make your own decisions.

That's great.

But that sort of brings with it, I guess, a low level of, tech alignment.

And it's also gets a lot harder to share code across these things.

It's harder if people need to, developers need to shift projects and work out how this new thing works.

But it wasn't, quite as simple as that.

We also had monoliths.

They were the opposite.

There was a, high level of alignment in terms of the tech because it's all in the one code base, so it's a little bit easier to make sure we're all using the same thing.

But again, if you are forced to go into other teams' code bases to make changes and to build out your products, then your autonomy takes a real hit.

So that wasn't ideal either.

So I think really what we were after was that happy medium, right?

If you think of it as a bit of a pendulum, you've got your monolith over there on the sort of far left-hand side.

You've got high alignment, fantastic, but low autonomy.

And then on the other side you've got the, small apps teams were highly autonomous, fantastic, but they had a low level of tech alignment.

So how do we get to that middle point?

What's the secret source?

So it was more so around, very much around how do we actually enable teams to spend most of their time solving those customer problems and not just doing all the other stuff?

I think most of us who are working in this area had heard about micro frontend architectures before.

I think nearly everyone probably knows of the the 2014 blog post around microservices from Martin Fowler and James Lewis.

One of the things that came outta that was that they predicted, that eventually front ends would be partitioned to match those backend microservices.

And that was really the sort of the beginning of the idea of, micro-frontends.

And now it's evolved very much to quite, quite a substantial movement.

It's if you do a Google for micro-frontends, you're bound to get a lot of results.

And the, whole concept is that you can basically have a website that's formed from a whole bunch of smaller parts that are composed together to form that site.

And it's a very powerful concept.

It's a lot harder to build in, practice, but that's all right.

We also spoke to our digital partner who's ThoughtWorks just to get a better picture of what the rest of the industry were doing.

Were we crazy to think that this might solve our problems?

And so we came outta that even more confident that this was, this is what we needed.

And we also looked at the whole do we just buy this thing off the shelf do we need to actually build this?

And there weren't a lot of options back in 2019.

There were definitely some, but they all came with some downsides that we weren't really willing to to take on board.

A lot of them were quite generic, so they supported a whole lot of front end techs.

Maybe it was React or Vue or, whatever.

And that was great.

We could probably take one of these, we could run with it pretty quickly.

But ultimately, having a lot more control over it and making it specific for how we needed to develop was gonna give us a lot more a lot more bang for buck.

So we eventually decided that we would go ahead and build our own.

First thing was the definition of a micro-frontend for us.

You may have heard, if you've ever read up about micro-frontends, that there's, talk of like horizontal versus the, vertical split which I'll get to in a second.

We could've gone and just said the whole page is the micro-frontend.

That would've given us a whole bunch of benefits, but it wouldn't really have helped that, header footer problem that I was talking about before where they just get updated in a few different places.

So then I guess it was like perhaps all of the components on the page are the micro-frontends, and this is like that, horizontal split that people talk about.

At, runtime, for example, you could have one of these, imagine these as being products owned by different teams.

You could have one of these things updated somewhere and then just auto-magically updated almost in real time across your site.

That would've been fantastic, and that may well be somewhere that we head to in the future.

but really for us our teams were very much structured around the content that resided between the header and footer.

So for us, this was our definition.

And, this was the, I guess a vertical sort of view of a micro-frontend 'cause it encompasses most of the page.

So it really.

We still like the idea of, teams being able to build their own apps.

We still wanted to give them that high level of autonomy have their own git repo, have their own apps in there.

We just didn't want them to build the entire app.

We really just wanted them to focus on what product really cared about, which was everything in between that header and footer.

So we started building this thing.

This was probably the easier part.

There was a whole bunch of capabilities.

So this is like that repetitive plumbing the integration to the analytics Server, for example.

These were all built out as a bunch of libraries that could be used within our platform.

But just as importantly, a lot of these libraries we built to use on our legacy apps as well.

And that was actually quite important in the end because in terms of talking about return on investment, if you can talk to the people that hold the purse strings and say, this thing we're building is not just for the future, it's something we can use across legacy apps as well, then that's really beneficial.

So, we built out a whole bunch of these libraries that can be consumed by our micro-frontends.

And I'm sure developers were pretty happy they didn't have to build yet another integration to our analytics servers.

As much as that is very fun.

But I think the most compelling part really for us was the actual runtime itself.

To use a few buzzwords it's a effectively just a universal single page app.

So it gets loaded once out into the browser, but that same code that runs there also runs on the server.

So we, we very much support ser Server Side Rendering, which is, it's something pretty important for us in terms of the, SEO benefits.

And because we provide a runtime, it's almost like a platform as a service for us.

So just starting out in in browser land again, you can see here that the, little purple bit there in the in between the header and footer, again, that's the part that the actual development teams, the product teams really have to worry about.

All of the the light blue all the capabilities, the actual runtime itself, you know how this thing gets loaded and rendered onto the page.

That's all provided by our micro-frontend platform.

Of course being universal, it does render on the server as well, what you're seeing over there on the right hand side, like what's in the boxes isn't too super important, but it's effectively one run time, which is built and managed by the one team, and it's responsible for powering all of the micro-frontends.

So as you can imagine, that is a relatively complex piece.

We've done a lot of work around making sure that micro-frontends run in isolation so that they don't or we minimize the chance of them impacting other micro-frontends.

Before, when teams were building and deploying their own Web apps, if their Web app went down, that's fine.

It wouldn't impact the other ones.

We needed to ensure that we kept that same level of resilience with this.

But what's really fantastic is, that with the runtime as it is now, there's there's less operational overhead.

There's fewer DR-disaster recovery plans, there's less bespoke infrastructure to run.

And the tooling was really important.

We provided tooling straight off the bat for teams to build and deploy these micro-frontends.

And that's very important.

Just because in terms of we wanted developers to want to use this, right?

We didn't wanna have to go out there and bang a big stick and say, you have to because the teams aren't deploying an entire runtime now they're just deploying bundles up into the cloud.

Deployments are super, super quick which is just fantastic.

You can talk within seconds and actually have a micro-frontend deployed out there in front of the users, And of course, none of this really would've been that successful if we hadn't have, again, focused on that developer experience.

So the documentation and the support for us over Slack was really important.

And we made sure that from the time we had our first sort of 'customer' or sort of team building a micro-frontend on board, that we had documentation ready to go.

We had support, we had a whole team there to, back it.

So in terms of benefits yeah, we've absolutely moved the dial.

We've now got quite a few teams.

I think there's well over 20 different micro-frontends now running across REA and we are currently moving our core search experience onto this platform as well.

These teams are certainly spending a lot more time solving customer problems, which is really fantastic.

And these these systems are healthier too.

In terms of those system health criteria I mentioned before.

A lot of those get answered for free and they're provided by the platform.

So these systems are just inherently healthier.

But there's some other benefits, some obvious, some not.

Performance if we wanna roll out performance improvement let's say going back to Core Web Vitals, if we rolled out Brotli compression across our, stack and all the micro-frontends got that for free, instead of having to go across individually and, update them.

We rolled out a new type of CPU up in the cloud.

And again, all the micro-frontends just got that for free.

Same thing for security.

Rolling out patches is super easy 'cause there's only sort of one stack, one run time that we need to worry about.

And with features we can roll out new capabilities quite easily.

We can certainly roll out a new header and footer a lot more easily now because that is evergreen and it's just one place that it gets updated in.

And alongside that, we, luckily we had some good delivery data, so we were able to crunch some of the numbers on, how things were after, the fact.

And we were finding that teams that were building new products were able to build them with around sort of 50% of the time to get to market, which is a pretty staggering number.

Probably only bettered by the one next to it, which was the actual effort.

So the developer effort to build those same experiences was only around 25% of what it was previously.

And something that really makes developers happy was the ongoing custodianship to maintain them moving forward was only 10%.

So some, huge wins there for us.

So I guess just some, key takeaways.

And again, look this is what worked for us.

We're not saying these takeaways will be applicable for everyone, but I'm happy to talk about this for anybody who wants to chat later.

It's really around enabling growth of, autonomous teams, scaling them out, but still having a really strong technical alignment across the board.

You know, that's, that was really the thrust of why we did it.

You certainly have to be a certain size, I think to, see that as a benefit, but for us, we were absolutely there.

Think about how you're gonna measure your return on investment.

It's it's really hard to do that after the fact if you don't have any of that delivery data to work out what are we getting out of this?

When people come knocking and saying prove to us that this thing's actually worthwhile.

Having that delivery data that you can then show them before and after is just super important.

And treat your platform as a product.

It's a phrase that most people have heard of.

We've got a team around ours.

We, we've engaged early and very often with our users-that is the developers.

And we want to make sure we built something that they wanted to come to as opposed to having to push it out there.

Have a technical product manager have a full team around it to actually give the thing the best chance of surviving.

So thank you very much.