Moving fast and NOT break things: Experimenting in zero failure environments

Building new products is incredibly hard, especially if you want to do so quickly with few resources. All the great lean approaches developed over the past few years rely on fast experimentation and learning from failures.

This is great in consumer products but how can we do this in Enterprise SaaS where we’re often working on mission critical solutions that can never be allowed to fail? What if you’re building tools for industries where people’s lives are at risk if your product doesn’t work correctly the first time?

This was the exact challenge we faced at PlanGrid when looking to launch a brand new product for managing construction projects. Come hear how we got a very small cross-functional team to explore, build and start earning multi-million dollars in revenue in a matter of months.

Moving fast and NOT break things: Experimenting in zero failure environments.

Tom Alterman – VP Product and Design at StructionSite

Web Directions was the first conference Tom ever attended! Opened his eyes to an incredible community of people who share their ideas and support each other’s careers and he can truly say that it fundamentally changed his career path! Very honoured to share some learnings with us all.

First time he’s ever been able to do a talk without shoes but he did put pants on just for us!

Tom is going to discuss how to use lean development practices in a world and an industry where failure is never an option.

Tom is VP of Product and Design at a construction tech startup called StructionSite. They use computer vision and machine learning to make it easy to see what’s happening on your construction site. 3D imagery is turned it into a Google Street View style experience of your project. You can take temporal views and data analysis to measure what’s been done. Also means you can work remotely, which given the pandemic has been a huge benefit.

In 2015 Tom was involved with tech startup PlanGrid, which recognized the revolutionary potential of iphone and ipad capabilities for the construction industry.

Prior to this, all of the software related to construction was around building design and finance management. PlanGrid was the first product that could actually be used on site. Build was going to shave hours if not days off someone’s job and make them fall in love with their ipad.

This led to becoming the highest rated construction app in the app store and being used on 1.5 million projects in over 100 countries every day.

In 2018 PlanGrid was acquired by Autodesk for 900 million USD.

This talk is about one of the hardest things Tom learned how to do there; how to ship great quality products in a world where you cannot fail.

Tom recommends The Lean Startup by Eric Ries as a text which has fundamentally changed how products are designed and built over the last decade, and how companies themselves are designed and built.

A core tenet of the book is the idea of experimentation- Fail fast, succeed faster

Exemplified by Linkedin Founder Reid Hoffman’s If you’re not embarrassed by the first version of your product, you’ve launched too late and Zuckerberg’s Move Fast and Break Things mantras.

Tom loves the mantras and wanted to use them in his work, but in construction, nobody wants to discuss breaking or failing. Their livelihoods and lives literally depend on reliable tools. This culture is risk averse, but processes are often inefficient. Construction is a high stakes industry, but many of us are designing and building products which users rely on, and they do not want to be your guinea pigs for experiments. So the question is how to solve?

Here are the TL;DR key learnings for this talk:

  • Build a x-functional team that cares about the customer
  • Learn about the problem together
  • Treat everything as an experiment – not just what you’re building but also the processes and roles involved.
  • Do the least amount of work needed to test your hypothesis – be lazy!
  • Use humans to power the first functional MVP – particularly when in zero fail environments.
  • Storytime! Story One: Submittals.
    Submittals are an insane part of the construction process. Every material and process you plan to use needs to be approved by the architect before you start, which leads to paperwork and regulatory headaches.Everyone hates submittals – they involve detailed spreadsheets, piles of emails, and mistakes cost not only money but can cost lives.

    For example, in the 70s, a hotel causeway collapsed because one diagram was missed in a submittal, resulting in the death of 114 people. Submittals matter.

    PlanGrid’s mission was to make the submittal process better, or at least to make it not suck.

    As a scrappy startup, they had significant constraints; only four engineers and six months to complete the project, oh, and the future of the company was at stake!

    These constraints meant they needed to get really experimental. The process involved learning about the project together – they did tens of customer/user interviews and site visits to various construction projects with their whole team so all were simultaneously up to speed.

    Really focused on getting to know users – watched them doing submittals, which may have appeared mundane but through this process they learned the most about what they needed to fix.

    Process: go on site, learn, and write down all questions, then vote as a team on what they thought was most important to learn next, and rewrite the script for the next set of interviews to shape accordingly. This gave confidence in the solutions.

    It was crucial that the team as a whole was involved in this process, including engineers, much more important that everybody understood what they were doing and why than writing code quickly. Better to be understaffed than under focused.

    Hypothesis and Experiments – They did not have time for the traditional build loop of idea/build/launch/learn/reiterate. Needed to use a leaner process so went straight from idea to learn. Followed the same process from research – listed all ideas and ranked them in terms of risk.

    This identified the most important things to tackle and find the cheapest and quickest way to derisk that particular assumption and hypothesis. Then just a process of going over the list until they had full confidence in what they needed to build.

    They used tools like surveys, clickable prototypes for users to explore and give feedback on. these worked well, and left them with a big assumption that needed to be tested.

    Can we provide something more useful than spreadsheets and email? Hard to validate through dummy data – needed actual data on actual projects.

    This led to the concept of Concierge MVPs. How to validate whether something is useful without building it?

    Example of the cartoon character Bender from Futurama who refers to humans as meatbags. They called this part of the process ‘project meatbag’. Humans would be the machines, managing the first iteration, then replace themselves with code as they learned. Emailed five target customers offering a new submittals tool and asked them to send all their project related emails and spreadsheets in exchange for a weekly progress report. The ‘tool’ at that stage was their project team, the ‘meatbags’.

    The first iteration of the product was a simple Google spreadsheet that they manually updated, then added automation, conditional formatting and data validation to winnow down what mattered most and how to best represent it.

    Also set up dashboards, so first actual code were excel scripts to offer a bird’s eye view. Very simple product but delivering real insights to customers. For example one customer thought there was a mistake in the software as it showed some submittals running six months late. The data was correct, the submittals had been overlooked and were unknowingly costing customers tens of thousands of dollars per day.

    Also learned that customers needed an easy to read report that they could send to architects and shareholders. Used a simple photoshop template customized with their logo and emailed weekly.

    Key insight was that spreadsheet, dashboard, and the report was all that were really needed to create a fantastic submittals product. Competing tools had hundreds of other features but many were superfluous. Yes customers loved having their job made easier, but they were building a software product, not a concierge service. Standardization was important and customization was minimal.

    Results were that in four months, they tracked over 600 submittals, including a half a billion dollar infrastructure project in California, which their solution saved millions of dollars on. So they built fast, but reliably. This led to unexpected opportunities, as evidenced by…

    Storytime! Story 2:
    People in construction are used to mundane and tedious admin processes and tend to be unaware of the potential of tech to help them – so how to understand their true pain points

    Key question to hit on the problem: If you could clone yourself, what part of your job would you have that clone do? One thing that came up multiple times was the process of actually creating the list of submittals in the first place – this was a question they would never have thought to ask about. At the start of every project, someone on a construction site has to read often very lengthy specifications document, and extract from it every submittal that is required. This time consuming and tedious project seemed to engineers to be a problem that could be relatively easy to solve via machine learning. But this was not part of the original scope of work and so time and resources were limited. How to test value prior to energy and time investment?

    Enter the meatbags again! Used a similar process – ‘human powered machine learning’ – wrote a human readable script that a layperson could follow and that would be easy to implement via a simple machine learning algorithm. Then reached out to customers, then hired data entry contractors to follow the script. Cheap solution and also a win win – offered excellent product validation and great training data for machine learning. They learned that the process did not need to be super fast, and in precision vs recall it was far more important to capture all possible submittals, even those which may not be required, because customers far preferred to prune a spreadsheet than risk missing any critical information required. All this feedback data meant they could also build the tool much faster.

    The conclusion to the above two stories is that they built a submittals tool in six months from start to finish with only four engineers. This tool was a relatively cheap add-on to their core product, but it earned the company an additional 5% in revenue that year.

    This also landed them on the list of the 13 Most Interesting Advances in Construction Technology of 2018. Another advance on that list was a robot that builds walls – a complex piece of technology solving a relatively simple problem, in contrast to PlanGrid’s tool which implemented relatively simple technology to solve a relatively complex problem, and did so initially through having humans imitate machines. But not everything went according to plan…

    What we messed up:They did not have an exit strategy for the concierge MVPs to automate the engineers out of the process, which cost a lot of time. They also deviated from their initial strategy of having the whole team involved during the discovery process to pulling the engineers out to focus on code in the testing phase. This led to an empathy gap with engineers not fully understanding customer pain points. This again cost valuable time. A further error was waiting until the end to bring sales and marketing teams into the process, leading to confusion and difficulty in explaining the value proposition.

    Recap: Build a great x function team that cares about the customer and learn together. Treat everything as an experiment, including roles and processes, do the least amount of work needed to test each hypothesis, and where failure can never be an option, use humans to power the first functional MVP.

    Experiments come in many shapes and sizes. Their process was originally applied to a new product, but since then it has been applied to existing products, products with thousands of users, and internal company processes. You can always be experimental.

    Feel free to reach out to Tom at via his website or on Twitter

    He is offering free coaching sessions to people just starting out in a product career, get in touch if you are interested!