GraphQL and The Web: Thinking beyond the technology

(bouncy music) - [Ben] Hey, so I know that John and the Web Directions crew work pretty hard to make this feel like a safe space, so I feel comfortable sharing with you a dark secret from my past and that is that I was a Java developer for ten years, the start of my career before I shifted to JavaScript.

But no judging here.

It does mean that I've been a server side developer as well as a frontend developer so I think that puts me in a really good position to share with you some of the things that backend developers secretly say about frontend developers when they're not around. (audience chuckles) "Can't those frontend developers just use "what we've given them?" "Can't those frontend developers just decide "what they really want, once and for all?" I think the implication usually being, we'll build it and then we never have to talk to them again. Then things start to get personal.

"How hard can it be to build a frontend really? "They're just tweaking CSS and bumping pixels around. "I mean actually, what do they actually do with all "of their time?" (audience laughing) But, let's be honest, frontend devs secretly say stuff about backend devs as well.

"Can't they just give us what we want? "What is this thing? "How am I supposed to use this? "How am I supposed to know everything I'm gonna need "in advance?" And, of course, "How hard can building a backend really be? "Aren't they just getting data out of a database? "What do they actually do with all of their time, "those backend devs?" I'm guessing there aren't too many backend devs in the room right now.

And even if there were, I probably wouldn't have much to say to them 'cause I've thought about this for a long time and I've kind of reached the conclusion that frontend devs really need to build their own servers. And I also think that GraphQL is a great way to do it. So in this talk I'm gonna kind of argue that case and we'll do it by talking about the problem first, and then the causes, and then we'll get into the solutions. All right.

So let's start with the problem.

I think that when it comes to loading data from backends, our client-side code, it's really doing too much stuff. And the end result is bad user experience.

Users end up with apps that take a long time to respond to what they're doing.

And also frankly, it's a bad developer experience. It's a bad experience for us as frontend developers 'cause our code bases tend to end up in a pretty bad sort of shape.

And I think there's some conflicting forces that lead our apps to this state.

And it usually starts with the realisation that a frontend developer has when they are given a new requirement and they have to do it.

And then there's, "I need a new piece of data." This is where it starts.

And faced with this question, a frontend developer usually has two options, they can call another endpoint to get that data or they can get more data from an existing endpoint. Each of these has its consequences and its trade offs. Let's drill into the first one, calling another endpoint.

And this is...

If you're a bit introverted like me, this is a very appealing one cause usually you don't have to talk to any other human to do this. You just know that there's another endpoint and it's got what you need so you call it, which is great. But there is a consequence to this and that is that now maybe in response to some user interaction instead of just calling one endpoint, you're now calling two.

Which mightn't sound too bad, but then someone comes along later on with another requirement for data.

And so maybe they call another endpoint.

And then someone comes along later, perhaps they didn't even know that these other endpoints have been called. I've been on really big, thirty developer code bases, where people just don't really even realise this stuff's happening and it progressively gets worse and worse.

And also, business logic tends to accumulate on the client when we do this.

Especially if you're dealing with very disparate set of backend systems.

All of this stuff ends up in the client.

So that's the first scenario.

Let's drill into the second scenario, getting more data from another endpoint.

So let's just say hypothetically, that you're a more extroverted developer, maybe you even have a friend who's a backend developer and so you go up to them and say, "Hey, I've got this new data that I need to get. "I'm calling this the existing endpoint, "could you just add this data to that endpoint for me?" And your backend developer friend goes, "Yeah sure, I'll do that." And so now your data's coming back and you're all good, which is fantastic.

Until you compound this decision over and over again, over time.

So someone else comes along later and ask their backend developer friend to add this data to the payload that's coming back.

Someone else does it later, perhaps they don't even know that other people have done this in the past. Perhaps there's even other clients that are calling this same endpoint and they're requesting their own data.

So this progressively gets worse and worse and worse end up with really fat payloads coming back many of which, the client only needs a small part of what's in the payload but all of this data's going over the network anyway. Your app still needs to process all of this data. And I think that whilst fat payloads are probably preferable to excessive chattiness in apps, it's a pretty poor trade off, when you think about it.

So let's just dive a little bit further into root causes. So chattiness is a problem because of latency. And latency comes down to the speed of light like it's a thing, it's not getting any faster basically (laughs). And so it's something that we have very little control over it, not really gonna change much.

But on your fast office network it mightn't be apparent to you that the speed of light is a thing but if you're out in the field on a spotty mobile network or if you're in a developing country, they don't really have that same luxury.

Next kind of root cause is often bandwidth or lack thereof. Now this is something that probably there is more scope for improvement and it is improving but again, if you're in the developing world it's certainly not a given.

Really we wanna pass as little data over the wire or the network as we can.

So both of these causes they're pretty intractable, like they're physical things, there's only so much we can do.

So are we stuck? But I think there's this kinda third thing that we don't really talk about very much and that's our culture as developers and where it's kind of gone.

'Cause basically what we're doing these days is we're taking this big, slow, sometimes unreliable network, it's also known as the Internet and we're using it to partition people into two groups, frontend developers and backend developers. And that has lots of consequences for us and I think it's getting worse as our client-side apps get more complex, this partitioning. Because once upon a time, when we were returning HTML from servers frontend people were backend people but that's not really the case anymore.

Now writing client-side code can be a full time job. You can be a full time JavaScript developer running just code for the browser or an iOS developer or an Android developer. So it can be tempting for us to kind of abdicate our responsibility to the user.

But we're the closest to the user at the end of the day, we're responsible for their experience as frontend developers and we're also closest to our code and the state that it ends up in.

Backend devs frankly probably can't, maybe even shouldn't be expected to care as much about this stuff as we do.

So I think that we are the ones that should have the final responsibility for how our app performs over the Internet, not just on the client-side in the browser or on a device but over the Internet.

And really the only way to achieve this is that we need to build our own servers.

Now I don't mean the whole stack all the way down but I just mean maybe a little server, sort of the first point of contact for your app. You need to be writing a client as well as a server. Now some of you are probably saying, "Hey Ben, there's already a pattern "for doing this sort of thing, it's been around for ages. "It's called the BFF pattern, "the backend for frontend pattern." Or if you've got an eight year old daughter like I do, this is kind of the first thing that comes to mind. And in a sense, best friends forever is kind of appropriate 'cause with the BFF pattern your backend and your frontend, they really are best friends forever.

We're talking about a backend that's been designed and built exclusively to meet the needs of a particular client and usually it's UI developers who are building these backend for frontends and that's fantastic.

And it's a step in the right direction, the BFF pattern. But it's held back by something and it's something for now that up until recently we've kind of taken as a given and inevitable.

And what I'm talking about is REST.

We really need to talk about REST.

I'm just gonna come out and say it.

I think that as an architectural pattern for communicating between highly interactive clients and servers over a network like the Internet, I'm not sure that REST is really the ideal anymore. It might've been once but I just don't think it is anymore especially in the mobile world.

And I'm just talking about this very specific use case to be clear though.

Complex clients communicating with a backend over the Internet.

And I guess the other important point I wanna make is whilst in theory REST could be, in practise it's really not.

And I'm not sure whether I really wanna wait around any longer to see if it gets there. I think there are two main main issues that I have with REST versus query strings.

So with REST we use URLs to refer to resources. The meaningful part of the URL is usually the path and the query string.

The path will give you a way to address some entity but really if you want any flexibility, it comes down to what you do in the query string. So for example, if you need to filter or sort data you gotta find a way to do it with a query string. If you wanna opt in to requesting complex nested data structures without having the server return data you don't want, you gonna need to figure out a way to do it with a query string.

If you wanted to opt in to complex server side calculations without having multiple requests, you're gonna have to find a way to do it with a query string.

Now I'm not sure if you guys know this but it's actually possible with REST to specify content type, not just with JSON but you can provide a schema in a content type that specifies exactly the data that you're gonna wanna get. Does anybody know about that? And I'm guessing no one's done it either.

Neither have I.

So all of this stuff is in theory possible with REST but it just doesn't play out that way in practise. Usually just comes down to query strings.

A second issue with REST is response structure. REST is usually response agnostic and frankly the structure of a JSON is usually just an afterthought and it usually is JSON these days.

Have you guys ever worked on a project where basically the only way that you could ever really know what a server was gonna return to you was to actually call it, probably in production? I work on those projects all the time.

Have any of you ever worked in a project where there's supposedly been Swagger put in place to tell you all of this stuff? But at the end of the day, the only way you really know what a server's gonna return to you is to do it in production.

And I think the thing here is that while Swagger and other frameworks like it are great, they kind of bolt-ons to REST.

They came after the fact.

They're not foundational, we're on the wrong foundation here.

And I take a little bit further and say that REST is based upon this idea that we use discrete endpoints to address objects. But our data is more complicated than that now, we've got inter-relationships.

We've really got a bi-directional graph and we need something that lets us traverse that graph easily and efficiently over the Internet.

So as you can probably tell, this is kinda leading up to GraphQL.

We'll start with some of the technical properties that it's got.

So when Facebook kind of devised GraphQL, I think that from very early on it was designed to help with these issues of latency and bandwidth, these fundamental issues.

Clients can get all the data they need in just one hit. They only get what they asked for, don't get anything more. And I think also it's easy for the developers to evolve their data needs over time, that's fantastic. So GraphQL does this with two mechanisms, it's got it's own query language, it's very flexible. So if we're to take as an example, just the notion you of having customers and each customer having orders.

Here's a query that would get just the first name and last name of all of your customers.

Here's one that gets just the first name of all the customers that have a particular last name. Here's one that gets the first name and the date of all of the orders that your customers have. You can even work bi-directionally so what we're doing here is we're saying, get the first name and last name of the customers who made orders on this particular day.

It's a very flexible query language and it beats the heck out of using an ad-hoc approach with query strings.

Second big part o' GraphQL is really stems out of, well how do we know what to query? That's what we have schemas for.

They specify exactly what you can run against a server and they define the structure of the data and it's inter-relationships.

So here's kind of a schema for the queries that we just took a look at.

And a really important point here is that this schema language, it's typed.

Okay, so it support things like inheritance and enums and custom scalar types like dates if you need them, even things like polymorphism.

And that's actually really, really powerful 'cause GraphQL server libraries can pretty much enforce out of the box that your responses match these schemas.

And really importantly, the queries, the schemas, the types, they all call to the specification.

Basically if your server doesn't have a schema, it's not a GraphQL server.

This is a big difference from the way that things work with REST.

So you can probably guess I'm a big fan but it's not just about the technology.

There's perhaps something else that people don't talk about as much and it kinda stems from the fact that I don't think it's that hard to write a GraphQL server. It's so easy in fact, that I think even we as frontend developers can do it, who would've thought. 'Cause you see, the thing is that the GraphQL server implementation, the reference implementation, it's built in JavaScript and runs a Node.

Apollo server which is probably, the most well known server library.

It's built in JavaScript, runs with Node.

You don't even have to build a server if you don't want things like Amazon AppSync, you can just wire a schema directly to data sources. So I guess what I'm saying is that if you're already running JavaScript as a frontend web developer.

It's a bit of a no-brainer that you would use JavaScript to write this server as well. And in my experience once you've nailed the basic patterns for writing a GraphQL server, it's pretty straightforward. So if we're to accept the premise that we should be writing a server but doing it with REST just seems like too much work, I would seriously suggest that you consider using GraphQL to build your server, but this is gonna require a little bit of a change in mindset.

Perhaps most significantly, over time we've come to equate frontend and backend as being the same as client and server.

We can't do that anymore.

We really need to consider our frontends as extending down into the server.

The server as being an extension of a frontends. That's really important.

Now this makes some frontend developers rather nervous. I get asked lots of questions about this.

Here are probably the top three: "Does this mean I have to own the whole backend?" No, cause a GraphQL server shouldn't actually do that much. It's main purpose for existence is just to aggregate services into a schema that makes it easy for a client to efficiently query and get the data that it needs.

That's its purpose in life.

What I found in practise also is that it's a good place where you can paper over all of the crazy weirdness of your myriad backend systems to present that one unified schema to your client. Put bluntly, a GraphQL server is great place to bury bodies.

It's a much better place to bury bodies than the client. And GraphQL servers they don't have to just interact with, REST services, they can interact with databases if they need to, they can interact with AWS services. The project I'm on at the moment, our GraphQL server interacts with Kafka and that is really something you don't wanna do from the client-side.

Next question that often comes up, "Should the same developers work on both the GraphQL server "and the client-side?" Yes absolutely, preferably in the same PR.

There is an initial learning period and there's a bit of terminology you have to get your head around but I found that especially if you're using JavaScript, there's just not much of a cognitive jolt for devs moving between the client and the server.

It's important to realise that if you split this work up, you're basically splitting your team's ownership of the user experience, which is not a good thing. It's how we kind of got in this mess in the first place, because to avoid having to wait for somebody else to do something, developers tend to cut corners, like we kinda talked about at the start.

And it also just splits the skill sets in your team which is never really a good thing.

Last question, "Does this mean I still have to talk to backend developers?" I'm afraid so.

'Cause all that we are really doing here is shifting stuff from the client to the server.

I think it brings us to a better place to do that but yeah, you're still gonna need to talk to backend developers.

And just as a follow up and people go to me, "Are the backend developers gonna "be territorial and not wanna cede control of the backend "and get angry about this?" And in my experience they really don't care because frankly writing a REST service that aggregates calls to other REST services for the sake of some tin-pot client that they really don't care about, it's usually not high on their list of priorities. They're quite happy to delegate that to somebody else. All right, so let's wrap this up.

I think that the combination of hard physical factors like latency and bandwidth plus this kind of culture that we've fallen into of separating frontend devs from backend devs, it's kind of led us to a bad place.

A place where we end up with poor user experience and also poor experiences for us as developers. As a consequence I think that really the best way is to step things back a little bit and go back to writing our own servers.

Something we've done in the past but many of us have kind of got out of the habit of doing. Things like BFFs are step in the right direction but they kinda held back by the limited expressiveness of REST.

I think that with its structure queries, its type schemas and its pretty good JavaScript support in particular, GraphQL is a really good way to build your own backend for frontend.

And the upside for your end-users is gonna be faster and more responsive apps.

I guess a downside for you is that you are gonna be responsible for a server.

Might be a while since you've been in that situation and you're gonna have to learn something new, that's not necessarily a bad thing.

But there's also gonna be upside for you as a developer and that is that you can have a far more manageable client-side code base and I can attest for that personally.

So I think that in short, GraphQL kind of empowers us as frontend devs to take a bit o' control back of our destiny.

And I can't guarantee that it won't be bit scary and I really can't guarantee that backend developers still won't complain about it.

But I do guarantee that we'll probably spend less time complaining about them because we're gonna have more control of the things we care about, which is our end-users and the state of our code bases. Thanks very much.

(audience applauding) (bouncy music)