Embrace feature flags, unlock better customer experience and developer confidence

Have no fear of perfection. You will never reach it. That's a quote by Salvador Dali. I think he was a very famous actor. But with future flags, you can reach near perfect software releases, unlocking both better customer experience and developer confidence.

And that's what I'm gonna talk about today. I am Gishan, I'm also a GD like Lavi. I've also been a Docker Captain since May of 2024. I blog at gishan.com.np, that's my LinkedIn profile. You're gonna see that QR code later as well. I'll go directly into the agenda.

You can get the slide decks on that particular QR code. Firstly, I'm going to talk about what feature flags are, then a bit into why and when and how to use them. After that, a couple of real life use cases, usages. Then a bit into some feature flag best practices and how does it achieve better customer experience.

Towards the end, there is increased developer confidence. How does feature flag help with that? And some takeaways. So let's get started. Feature flags, basically from Martin Fowler's website, Pete Hodgson says it's a way to modify system behavior without changing the code and deliver new functionality to users rapidly but safely.

So the two main things here are the features of the system behavior is changing without changing the code. And then an aspect of safety where you can continuously deploy things, but it doesn't reach the customer. So what it essentially means is deployment is technical task, releasing it actually to the customers is more of a business or an operation task. I think that's what you should get out of from here.

So visually it basically looks like this. This is a screenshot from Flaxsmith. You want the chat feature, you turn it on, you get it on. You want the dark mode, you turn it on, you get it. You want the new design design 2.0, you turn it on like X amount or probably 100% of your customers get it.

That's feature flag in a nutshell visual. There are multiple use cases of feature flag. This also comes from Martin Bower's website. The main focus of this talk is going to be around release toggles. Of course, there are other things like experiments could also be run as feature flags, opt toggles, permission toggles, this and that. But this particular talk is mostly focused on how I have used feature flags or how we have used feature flags in teams for at least 12-13 years now.

That would be the main focus of this. Let's go around, yeah, 13 years back story of how how I was introduced to Future Flags. So the case was, this is an e-commerce website. We were doing cash on delivery and credit cards, like cards, like user cards, bank cards.

Then there was a need to add PayPal. And the team lead, the CTO and I were in a meeting and basically they joked saying, if we lose money with this PayPal integration, it's gonna be cut from a salary. It didn't really happen, but that's what the, the whole premise was about and the whole joke was about. Then the team lead basically told me a way to do it in a very safe way.

His suggestion or his idea was add a very simple if condition that says if the email address is your email address at the company domain.com, then you see PayPal on the checkout page, else you don't. And that simple idea basically blew my mind. And that's how I was introduced to feature flags, so I can turn it on or off like Sean Shell.

And we did it like that. I first only put my email address, then in the next release, I put the whole team's email address, three or four people. We tested it pretty heavily for three or four days, and then we released it to customers. This is not really a feature flag by the book. We needed deployments to actually change the behavior, but then this was my introduction to feature flags.

So now this comes to when do we use feature flags then? Before I proceed, how many of you have actually used feature flags? Wow, probably more than half of the audience, that's good. So the point being, when would you use feature flags? In my opinion, whenever there is even a small bit of risk, it's always good to reach out and use feature flags.

More on it, as I said, even if there is any small business risk, try to reach out and use feature flags. Anything that you touch with existing features or critical path or money, always go in and try to put it behind a flag. If anything breaks, you always have something to fall back to without needing to do another full on deployment.

Similarly, any mid-size or large size feature that takes a longer time to develop, that means it'll take a longer time to release, use feature flags. The point being, all your code is can be merged on top of master or main or whatever branch you use while the development is still going on and you don't face yourself with big conflicts in your code.

So that is another good use when you want to use feature flags so that it can be released when it's done without the need of being like every small change needs to reach all the customers. Now, why use feature flag? Let's go back. Oh, wrong. Around a year and a half, not strike.

19th of July, 2024, on Good Friday, someone releases a security update, takes down 8.5 million Windows operating systems, causing havoc in airports, 10,000 plus flights canceled, hospitals can't use their systems, basically everything in chaos. Imagine if they had used feature flags and released only to 1% of the customers, that is 85,000 or something, Windows operating systems.

So they would have caught that bug pretty easy, probably fixed it over the weekend, good Mondays, but no, they basically deployed it to all of their customers, around 8.5-9 million Windows operating systems. The whole world came crashing down probably like a Y2K in real life and boom, right? Not only that, their share price basically tanked.

In like three or four days, it has come up a lot. But yeah, 19th of July that happened and it's 25th of July. Yeah, 25th of July, basically they were trading around 300 or 400, they came down to 254. What I want to point here is, yeah, you should be able to manage risk properly and that is a very good reason to be using feature flags.

Again, the main thing here is limited blast radius. The same example, if they had only reached 1% or 2% of the customers, 160,000 Windows operating system would have been affected. They would have time to breathe, they would have time to fix it, they would have time to roll back because it is behind a flag. Unfortunately, that didn't happen and then you had 10,000 flights being canceled, people in hospital can't really use the systems and all of that.

So let's go into how to use feature flags. Of course, like many technologies, there are multiple ways to do it. But what I have learned or we have learned in teams in the past 10 or 12 years is to start small. Maybe as I said, like start with a simple if statement.

If the email address ends in your company email address or your particular email address, then you have access to this feature. Probably start there. Once your requirements grow, then you could opt in or the whole team could opt in for a Full-on feature flag management software. It could be your LaunchDarkly, Flagsmith, whatever, Unleash.

We use Unleash at where I work right now. So it could be anything. It's just a matter of choice. One thing that you might want to think is do you want to use a fully SaaS option like LaunchDarkly or probably a open source version which also has a SaaS offering. That is something that you could weigh in, but that is probably step two or three after you are a bit into the feature flag journey.

The main point here is focus on the feature flag mindset. It's not, I use tool X or tool Y, it's mostly like we have this mindset of we need to use feature flags if there is a risk, if there is anything that needs to be maintained, or if something takes too long to develop, the code still needs to come into the main branch.

I think that is the mindset that you need to have. Plus, we are doing this eventually to give good customer experience and we don't want the customer experience to suffer. So I think those are some of the important things that you want to consider when using feature flags. So this side. So this is a very simple example.

You could have a feature flag service and then you could have something like should limit portfolio analysis. I'll come to this. I work for simply Wall Street. We have portfolios that is basically stock portfolios. I can talk about it a bit. And then a simple check could be if your email address is in the feature flag, then you get to or your portfolios will be limited. If not, you get the unlimited portfolio that has everything.

It could even start with, as I said, if your email address ends in yourcompany.com, you have this new feature, else you don't. Going into some real life use cases, this is from one of my previous employers. By the branding, I think you know who I'm talking about. If not, they have a new saying like, got you looking or something like that.

So if you are in Australia or like New Zealand, you already know who I'm talking about. This particular feature was, I think, mid 2020. So before that, this fashion e-commerce website was basically selling shoes and t-shirts for shoes and t-shirts. You sell size 37 or 39 or size medium or large. $140, it's $140 for all the sizes, but that doesn't apply to things like fragrances, lipsticks wear a fragrance that's 30ml sells for $167, whereas the same fragrance of 90ml sells for double the price.

So this particular feature, we developed it and then we released it, it was done all behind feature flags. And of course, we did it gradually. The first release was only to us, So here, whenever the feature was complete, whenever the smallest thing was done, it would automatically go to production, but always behind a feature flag, right?

And how we released it was we first released it to our team and then gradually to the whole organization and 10 to X percent of customers. How we did that was, yeah, we went from 0 to 10%, looked at everything, looked at logs, looked at alerts, things looked fine. Then we went from 10 to 25, similarly 25 to 50, and eventually from 75 to 100.

All of that, I think, happened in a span of 35 minutes, yeah, around 35 minutes. But I think we were either too lucky or probably we did very good testing all over so that it happened in that span of time of like 35-40 minutes. We could get it from 0-100% of the customers. Having said that, all of the small things that went to production were tested as it went to production.

By the team or at least by the engineer who was working on it. And that helped us deploy it so smoothly within half an hour, like 45 minutes. In a website that would probably have like millions of users every day or something like that. Another real-life use case, this was the thing that I had the code of. So as I said, I work for a company that helps investors do long-term stock investing decisions.

And let's say if you buy things with something like stake or another, I don't know, CBA, you could link your portfolios with simply Wall Street and then once you link your portfolio, all of the data would come in. Now, this particular feature done last year was we want to limit what the user can see depending on their plans and of course this was also developed with feature flags, anything that was complete went to production.

And then it was tested within the team and later tested within the organization, then only released to customers. So how that happened was, yeah, we had flags that were on the email address level first of the people that were inside the company, and then we gradually same thing released from 0 to 10, 10 to 20, eventually reaching to the 100% of the customers.

Now, let's get into some best practices for feature flags. The first one being limit the scope of your feature flags. I have seen teams using nested flags or flag with too much of scope. Probably don't do that. Limit your feature, focus it on one particular feature.

It is a good idea to use the same feature flag on the front end and over the backend or multiple environments, but I think it's a good idea for the flag to cover too much or even have nested flags. It's easier to link one thing to one particular flag rather than having two or three things being nested.

And the other thing is release gradually. Probably this is the fourth time I'm doing the same thing. Test within the team, then the organization, and then only go to X% of the customers. The X% being how large your customer base is. Yeah, another thing I've seen with teams is they add the flags, they do the testing.

It's already in production. It's only in production for weeks or months, and they just leave it there. Leave the code there, leave the feature flag as is there. I think that's a pretty bad thing. And as a best practice, it's very good to clean up after you do something. So do your housekeeping properly. Remove the code, remove the feature flag from the system, probably archive it and keep it there for some time and then delete it, but do that.

Another important thing with feature flags is to have proper access control. You should know who has access to what feature flags, because some feature flags could be critical, as well as an audit log of when and who changed the feature flags, just in case if there is an outage or something and you need to do a postmortem, you know what actually happened.

Of course, with these things, you always blame the process and not the person. There needs to be psychological safety, but having a data log of exactly what happened or how did we get into this place would be even important for feature flags. Some other things that would be very good to follow would be follow a proper naming convention, tags with your team or anything that is created so that it's much easier to figure out what you are dealing with. In your feature flag management software.

Now, all this is potentially or mainly done for better customer experience. Let's get into how that would be realized. The first and foremost is the customer is probably the third or the fourth layer that is actually having a look at that particular feature. So that makes it already a much better customer experience for them. You're not using your production customers as testers, right?

So you have already done the feature, put it on production. It's behind a feature flag, you and your team have already tested it. Other layer of testing is some of the other people in the organization, and then only it is reaching customers. So it's already like the third layer. One thing that we used to do at where we worked is whenever we had some big or critical features, We would call for a test party. What do I mean by a test party?

It's basically meeting like 30-45 minutes. We invite anyone who is interested to come in and test this new feature. We would surely add all of them who are inside the organization to this feature and then they would test it, right? And if they find any bugs, they would report it to us as we being the team who developed it, we would fix them and then only release it to the customers.

And that was a very good practice that got a lot of things because sometimes what happens is when you're working in a feature and you are within that particular cocoon, you don't really see it at the outside. And if someone from outside comes in and tests it for you, they find other bugs that basically you couldn't see because you were inside the system itself.

Another thing for better customer experience is there's less chance of the customer actually getting an error in the new feature. Because of the same thing that is probably the third or the fourth layer that is actually testing this new feature. So it's probably a lot less chance than using your production customers as your testers for a new feature.

And yes, all of this like good customer experience surely equates to a great branding and goodwill for the company. With all of this, feature flag also enables higher developer confidence. The first one for this being, there's less code conflicts because every small change can reach your main or master branch.

That means things are much faster, things go into the main branch faster, it's already in production. Is it enabled or not? That is the part where the feature flags comes into play. So you don't have 1,000 line PRs or MRs lying there and then no one wants to touch it. No one wants to take the risk of actually trying to deploy it. If you have smaller 50, 60, 100 lines of MRs always going in every day things are like going through a CI/CD pipeline and then reaching the master branch, it's a lot easier to control those features.

And as developers, you can always test in production. Testing in production always beats X number of testing in staging because you have the real traffic, you have everything actually happening, you have the real online database, you have all the users there. So testing production, that is also something enabled by feature flags because the feature is only available to you or the team or, I don't know, five people in your team.

You can go in and you can test in production, you can look at the monitors and check the logs and check that all of the things look good, all of your or any of your alerts are not fired. Of course, when the actual traffic flows in, you will probably see Another layer of issues, but that's where you have the control of I just want to release it to 5% of the customers, look at it over the weekend, look at it over the weekend, look at all of my logs and see that everything is fine.

When I am happy, when I verify that things are fine, then I can go into 10% or 20%. You could always take that path. Yeah, even though you might not like it, but rollbacks are part of the process. So if things go south and things start hitting the fan, things start hitting the fan, you can always roll back and the process being you don't need to do a full on deployment process. You just go in in your feature flag manager software, change the 5 to 10 or 0 or whatever, 5 to like a lower number from 5%, you go back to 0 or 0.1% or even 0 and then only retest everything within the team.

And then you have much higher like developer confidence to release it again when you know what the bugs are and then when they have been solved. LaunchDarkly, one of the most popular feature flag software, SaaS claims that 45% of LaunchDarkly customers could recover from an incident within an hour, if not minutes. And if it was not for feature flags, I don't know how long it would take.

But with all of that, feature flag is no free lunch. One thing that it surely introduces is complexity where you had to only test one straight path now because there was fork. You have to test multiple paths. Generally, what I do is I have like two browsers open, one where the user is in the feature flag and one where the user is not in the feature flag.

And I make sure that both of the paths are tested so that I'm sure that both of these forks are working. Misconfigured flags can be damaging. I have been guilty of this probably more than I would like to be, more times than I would like to be. Sometimes I think that the feature flag is targeting this or this particular set of users, but it is not.

The point here being learn how your feature flag software works. Know that if there are five people in that particular flag right now, the feature is available to only those five people and not on and it's not disabled for those five people. So things like that, just be careful of what you're trying to do because, yeah, misconfigured flags, if you can't really get it to do what you want to do, can be damaging.

And rolling back a feature where there is a write, for example, like a set of customers have already seen that form and they have filled it and they have the data already being saved in the system, and now you need to roll back, that becomes tricky because if, like, 25% of the customers already had it and now you want to come back to, like, 5% they can start complaining saying, hey, I saw this yesterday, I don't have it now, and probably you might need to add that particular customer to the feature flag. But that's a tricky situation.

As I said, everything with technology, it depends and has trade-offs, and these are some of the trade-offs that you need to tackle with feature flags. The top three things I would like you to take away from this would be use feature flags, Optimally, whenever you need to use it, make the most out of it, but don't overdo it. As I said, nested flags, flags for smaller things, flags for things that are not really that risky.

So as most things in technology, be careful and use it optimally. Follow the best practices. I've listed some from my experience, probably there are more. I saw more than half of the hands up here, so other people might have other good best practices that they have encountered.

The main thing I remember would be always remember to clean feature flags. That's something that most teams and team members either forget to do or are too lazy to do. The point being of feature flags, it enables better customer experience and with that you also get higher developer confidence attached to it.

That's the main point of trying to use feature flags. You don't want to make it blast radius 100% of the customers. You want it to be 3% for at least three or five days. Be convinced that your logs look fine, your alerts aren't firing, everything looks good, things are healthy, and then only go to more customers. That's it. Thank you.

And if you have questions, I can take them.