Moving fast and NOT break things: Experimenting in zero failure environments
Tom Alterman–Moving fast and NOT break things: Experimenting in zero failure environments
(upbeat music) - Hi there.
Now I'm very sad that I can't be with you all in Melbourne and seeing all of your faces, but I feel truly privileged to be able to give this talk today.
Web Directions was actually the first conference I ever attended and it opened my eyes to this incredible community of people who share their ideas and support each other throughout our careers. And I can truly say that it fundamentally changed my career path.
So I'm truly honoured that I can be sharing some of the things that I've learned so far with you all today. Now this is a weird experience giving a talk from my living room.
It's the first time I've ever been able to give a talk without wearing any shoes.
But don't worry, just for you guys, I did put on some trousers today.
Now I'm gonna be talking about how to use lean development practises in a world and an industry where failure is never an option. To quickly introduce myself.
I'm the VP of Product and Design at a construction tech startup called Struction Site. At Struction Site we use computer vision and machine learning to make it incredibly easy to see what's happening on your construction site. We take 360 imagery and we turn that into a Google street view, like experience of your project. And so you can see what's happening today, this week, last year.
And we actually even analyse the images to understand automatically what's happening on your site and how much work has got done.
You also like to do your work remotely which has been incredibly valuable in a time of COVID. Now this isn't my first experience of construction. I actually fell in love with the industry back in 2015, when I joined a tech startup called Plan Grid. We were the first people to take the form factors of the iPhone and the iPad and understand this could revolutionise how construction was done on construction projects.
Up until then, all of the software that people had to build a building were really around the designing of the building or the managing the finances of a building back in the office. This was the first time that people had a product that with them when you were actually doing the work on a site.
And our founders really drilled into us, the notion of Build was going to shave hours if not days of someone's job, and make someone fall in love with that iPad. And they really did.
We were the highest rated construction app in the app store. And by the time I left, we were on 1.5 million projects in over 100 countries every day.
And in end of 2018, Autodesk acquired us for about $900 million U.S. dollars.
So this talk is about one of the hardest things I learned how to do there. And as a ship, great quality products in a world where you can not fail.
Honestly, this is just an excuse to show you one of my favourite gifts of all time.
I'm sure all, if not, most of you have read this book. If not, I really encourage you to reach out for it. It has fundamentally changed how products are designed and built over the last 10 years.
In fact, it's changed how companies are designed and built over the last 10 years.
And one of the core tenants of this book is idea of experimentation.
And as, Eric Ries, puts it, "Fail fast, succeed faster." I'm sure you've also seen this quote from, Reid Hoffman, the founder of LinkedIn, who says, "If you're not embarrassed by the first version "of your product, you've launched too late." And then there's Facebook's famous mantra of, "Move fast and break things." It's okay for things to fall over every so often as long as you're innovating and executing really quickly. Now, I really love these mantras and I really wanted to implement them at Plan Grid, but I had a problem.
And as I was building products for these guys, these guys never wanna hear about anything failing ever. Their literal livelihoods and lives depend on the tools that they use being reliable 100% of the time. And that's one of the real challenges in construction, that a lot of the processes are really inefficient and not very good, but no one wants to take any risks on doing anything differently.
The costs are just too great.
So people for a long time, we're just stuck with really bad tools and really bad processes to get their jobs done.
And that was a world that us as a scrappy startup, we're trying to change.
Now, not all of you are gonna be in construction or an industry as mission critical as this, but I'm sure a lot of you're working on products that your users need to rely on every single day. And so they never wanna feel like the Guinea pigs for one of your lean experiments.
So I'm sure all of you are paying incredibly good attention but for the very small minority of you who are like me and have terrible attention spans when you're watching videos online, I'm gonna give you the key learnings from this talk straight up. Firstly, you should build a cross-functional team that truly cares about the customer.
Then you should all learn about the problems that customer has together.
You should treat everything as an experiment. And I don't just mean the things that you're thinking of building, but the processes that you're gonna use as a team, or even the roles that each of you are gonna play on that team. Do the least amount of work you need to test that hypothesis.
And I really mean try and be as lazy as you can. And the really critical thing here if you're in a truly zero-failure environment is use humans to power the first functional MVP.
So to elaborate on all of that, I'm gonna tell you a couple of stories, all based around one of the most insane parts of the construction process and that's called submittals.
So what are submittals? Well, in the United before you can start any work on a construction site every single material you're planning to use and every single process you're thinking of using needs to get approved by the architect.
That means literally every brick, every door, down to the nuts and bolts, you're gonna use need get approved. Now this leaves 1,000 of documents that need to be sent out, reviewed, amended, approved before the materials can be ordered and the work can start. Now, if that sounds crazy, that's because it truly is. And there's two really important things to know about this. Firstly, everyone hates this.
Now that's because it is a endless amount of paperwork. You're dealing with huge spreadsheets to try and keep track of millions of emails. And every time you make a mistake that could cost you thousands of dollars.
In fact, it may not just cost you thousands of dollars, it can literally cost lives.
And this isn't a theoretical.
Back in the 70s, there was a hotel whose causeway collapsed all because this one little diagram was missed in a 100 page submittal, and 114 people died.
So when you say this is important this is an extremely important process.
So our mission, whether we chose to accept it or not was how to make it much better or at least to make it not suck? As a scrappy startup, we had some pretty fun constraints. Firstly, we're gonna have four engineers, actually we're gonna have two engineers and then another two were gonna join us in a bit. And we needed to get this from start to launch to something that people would actually buy within six months. Oh, and the future of the company is at stake. So don't mess this up.
This caused us to rethink how you're gonna do this. And we realise we need to get really experimental. So the first thing we decided to do was learn about the problem together.
We didn't have time to document things and create huge amounts of artefacts.
We needed everyone on the team to be in sync about what it was that we were trying to do and why. So what we did was we went on tens of customer interviews and visits onto actual construction sites.
And this was every week going out to a new site and spending a huge amount of time with the users. And here in this photo, you're seeing engineers, research design, QA, everyone on the project being involved. And we really spent time with our users.
It was really important for us to learn all of the things that they do and why they did them.
So we sat behind them and watched them do submittals. To be honest it kind of weirded them out that we wanted to see this 'cause it was a rather mundane process. But it was really in those mundane parts we learned the most about what we should fix about it. We go on site, we'd learn some things and then we'd come back, and then write down all the questions that we still had or that we hadn't seen enough evidence of.
And then as a team we would vote to see what was the most important things we want to learn next and rewrite the script for the next set of interviews to shape that. And we'd continue doing that until we learned all the things we needed to, to be confident about the solution. And everyone was involved.
We really mean this, in all, including the engineers it was much more important for us that everyone was bought into this and truly understood it than that we started writing code quickly.
And we found it was better to be understaffed than under focused.
We had a really great engineer who was actually more interested in systems work.
It wasn't fully bought into the process and so he went off to do other fantastic work in the company. We were much happier being an engineer down than having someone who wasn't fully bought in. So we had a good day the hypothesis about what we need to build.
Now we need to see whether we were right about it or not. So this is where we led to the experimental phase. Now that I'm sure you're all familiar with the traditional build loop.
You have an idea, you build it, you launch it, you learn and then you iterate as many times as you need to, to get that right.
Well, we didn't have the time for that.
So we need to apply the lean process of going straight from idea to learning and iterating on that as quickly as possible. So we followed a pretty similar approach to our research. We listed out all the ideas we had and we ranked them in terms of risk.
How risky was it that if this was wrong that our entire solution was gonna fail.
And once we had that risk ranked it was pretty easy for us to then see the most important things we need to tackle and then find out what was the cheapest and quickest way that we could de-risk that particular assumption and hypothesis.
And then we just go through this list until we felt fully confident about what we needed to build. Some of these were really easy to investigate. We could send out a survey and get some information really quickly.
Sometimes we bought clickable prototypes to allow users to really explore the product and give us much more detailed feedback.
And those worked really well.
But then we were left with this really meaty assumption that we need to test.
Could we build something that was better than the spreadsheets and emails that we're doing today? Now, this was incredibly hard to validate because the dummy data wasn't resonating in the prototypes. To get real feedback on this we need to see people using their actual data on some actual projects.
So this led us to what we called concierge MVPs. We needed something real that was working, that was actually managing the submittals on a project, but it needed to be reliable.
We didn't want any buildings falling down on us. So how do we validate whether something's useful without actually building it? I'm sure a lot of you are familiar with the cartoon "Futurama" and you might recall, Bender, and what he calls humans.
We call this project meat bag.
We would be the machines.
You would have us as humans managing the first iteration of this product and then slowly replace ourselves with code as we learned what we needed to do and make sure that it worked correctly.
So what we did was we emailed about five customers who represented a full array of our target audience and said, hey, we have this great new submittals tool just forward us your emails and your spreadsheets, keep us in the loop with all your communications and we'll show you what's happening and where, and give you some really high quality reports. In reality, that was us.
We each took a project and we were the machine that was actually managing this process.
And this is what the first iteration of our product looked like.
It was a simple Google spreadsheet.
We had a standard template, which we designed that matched the data that we were thinking of collecting and how we wanted to present it.
And then we would manually update the spreadsheet as we got new information into the system.
Then we started adding some basic automation, some conditional formatting, some data validation, all things to save us time as we learned what really mattered and how to represent that effectively. We also set up some dashboards.
So the first actual code we wrote were Excel scripts all to basically give some relatively simple overview of what's happening, what's overdue and that bird's eye view of what's happening on that project.
Now, this is still a really simple spreadsheet but to our customers, this was magic.
This was a real product that was delivering some insights that they never seen before.
Some of our customers actually said, "Hey, I think there's "something wrong with your product 'cause it's showing "some submittals that are over six months late." And once they looked into it, they realised, oh no, we do actually have some metals that are six months late. And that was causing them $10,000 a day in costs that they didn't even know they were incurring because they just lost this information in the crazy processes involved in construction. We also learnt that what they really needed was a really nicely styled report that they could then send out to their architect or their owner and the stakeholders that showed what was going on and what to do about it. So then we just had a Photoshop template which we would update every week with their logo and their information and send that out to them via email. And that's where we learned that the spreadsheet, the dashboard and the report were all that were really needed to have a fantastic submittals product. And that was ignoring hundreds of other features, which other submittal tools out there on the market included. We'd realised that if we built all of this, we'd have a product that the customers truly loved. An important thing to note here was that customers are gonna love you doing their jobs for them, but we were building a software product not a concierge service.
So we took great big lengths to make sure that whatever we were doing were things that we could easily replicate in code.
And that we were not doing too much customization, that we were offering a standard service that everyone saw the same results of and gave us the same feedback about so that we made sure that we could build this very easily. So what were the results? Well, in four months, we tracked over 600 submittals, including for this half a billion dollar infrastructure project in California. And we literally saved that project millions of dollars through these spreadsheets.
So this allowed us to build the solution incredibly quickly and know that what we were building was actually really gonna solve our customer's problems. Now the other thing that this did was actually opened up a whole world of opportunity that we never even expected to see when we started this process.
So that's what our second story is about, that new opportunity that we discovered.
And something to know about people in construction is that they are insanely accommodating people. If you ask, Ahmed, here, what do you hate about your job? He'll sit there and say nothing really, I love my job. And this is after he's just shown us some of the most mundane and tedious admin processes that I've ever seen.
And that's because they have incredibly hard jobs but they're just happy to get things done.
And honestly that they don't even know the world of things that technology can do to help them.
So it's actually quite hard for us to figure out what we needed to do to understand their true pain points. After a bit of experimentation, we actually landed on the question that was the most revealing and something I'd encourage you to use if you're struggling to understand how to really improve someone's productivity. We asked, "If you could clone yourself "and convince that clone to do the part of your job "that you love, the least what would you have them do?" And this really opened up the world to them to share the things that they really didn't like and for us to learn what are the things that we really needed to focus on. And one thing that came up time and time again, that we never even thought about asking them was actually creating the list of submittals in the first place.
There's a really crappy job that someone on a construction site needs to do at the start of every project, which is read a specifications document that's somewhere between three and 10,000 pages long. Now that is a process which they have to just read every single line and then extract from it every submittal that is required.
That can take two weeks of someone's life to get done. And it has to be done perfectly 'cause again, bad things happen if you miss a submittal.
Now, our engineers looked at that and said, "Hey that could be quite "a solvable machine learning problem.
"It might not be that to build." But we were already months into this project. We didn't have that much time left and we already had a very tight scope.
But this seemed like an opportunity that was too good to miss.
So how could we test whether people would find this valuable before we really invested any time and energy in it? Luckily we had a process and to the meat bags. So what we did was we sat down and wrote a human readable script, something that someone, without any knowledge of construction could follow to get submittals out of the specification. The other key thing about this is that we wrote it in a way that we knew would be very easy to implement via simple machine learning algorithm. And then we reached out to some customers and we said, "Hey we had this great new submittals tool "that automatically creates your submittal log for you. "All you have to do is send us your specifications "and we will send you back a spreadsheet "of all of your submittals." Now it's an experimental tool so it might take a few days to get back to you.
And then we just hired some data-entry contractors to follow that script.
It was really cheap and we learned an incredible amount. And it was really a win-win in two ways.
Firstly, the product validation was fantastic. We learned some things that just made it so much easier to build this.
Firstly, we learned they didn't need to be that fast. It was okay for it to take a few days.
The second thing is precision versus recall which is really important when you're building machine learning tools.
We learned that was actually way more important to get absolutely everything even if we caught loads of things that weren't actually required 'cause people were happily able to prune a spreadsheet. But you can never miss a single submittal.
What's more is that we were actually collecting labelled training data that made it so much faster to build the actual solution in the end. So we were winning, the customers were winning and the engineers were winning because we were just making it so much faster to get this built. So the conclusion to these stories is a submittal solution that we built in six months from start to finish with four engineers.
And despite this being a relatively cheap add-on to the core product, it was earning the company an extra 5% in revenue that first year that it shipped.
One of my favourite things about this was that we were included in this list of the 13 Most Interesting Advances in Construction that year. And were included alongside this robot that builds walls. And what I love about this is that this robot is a complex piece of technology solving a relatively simple problem for a person.
Whereas what we did was implemented some relatively simple technology to solve a relatively complex problem for people.
And we did it by initially having humans imitate machines. So we felt pretty smug.
Now this didn't all go according to plan.
So I wanna share some of the things that we got wrong so that you'll hopefully learn from, if you try and implement this.
Now one of the biggest things we got wrong is not having an exit strategy.
Our initial plan was that we were going to be doing these concierge Mbps for a couple of weeks as the engineers automated us out of the process. But it's a tech project.
Things happen, things were more complicated than we thought and we, you didn't get the engineers up to speed fast enough.
So we ended up having to do this process for four months and that was four months of effectively having two jobs. And that was particularly stressful for the person who was responsible for that half a billion dollar infrastructure project in California. I really just had to thank her every day for not quitting on us.
So if you're gonna do this think about what your exit strategy is.
What happens if it's taking longer than you expect to get the solution implemented? What happens if you actually explore this and realise this is not a solution you're gonna build, how are you gonna do right by your customers? The other thing was we did a really great job in involving all of the engineers and getting them bought in, in the discovery process. But once it got to the concierge MVP, we decided it was better for them to be focusing on writing code than it was actually feeling those pain points by managing those submittals.
And that's where the cracks, the empathy began. And there were a few mistakes that cost us a few weeks of development, which were pretty confident wouldn't have happened if the engineers actually felt, well, what it was actually like to manage submittals. So in retrospect, what we would have done was have them at least for a week or two, feel what it was like to actually hand crank those concierge MVPs. The last thing was that we didn't include sales and marketing until the very end of the project. And that meant it was a lot more confusing, a lot more difficult than it needed to be, to explain the value proposition, figure out how to sell and who to sell this product to.
And what we've done since is always include the product marketing manager at least in the research and discovery phases, which meant that the go-to-market process was so much smoother when we got there.
So just recap, if you wanna try this process at home, build a great cross-functional team that truly cares about the customer and all learn about the problems that customer has together. Treat everything as an experiment.
That's the product you're gonna build, as well as the processes you're gonna use and the roles that each of you are gonna play on that team. Do the least amount of work that you need to do to test each hypothesis, be as lazy as you can be. And if you're in an environment where failure can never be an option, use humans to power that first functional MVP. The last thing we'll leave you with is that experiments come in many shapes and sizes. We tried out this process on a new product in this example, but since then, we've actually applied it to existing products, products with thousands of users using them every day and even internal processes within the company. So always think about how you can be experimental whenever you're trying to approach a new problem. So that's everything from me.
Thank you so much for listening.
And please reach out if you have any questions or comments, or wanna have a further discussion about any of the things I talked about today. Another thing is that I am offering free coaching sessions at the moment to people who are just starting or are trying to move into a product career, as I know how hard that can really be.
So if you're interested in that, I have a couple of opportunities still open.
So feel free to reach out via email or Twitter. Thank you so much.
And I really hope to see you all in Melbourne next year. (upbeat music)