BETA
This is a BETA experience. You may opt-out by clicking here

More From Forbes

Edit Story

Five Pitfalls Of Running Lean Startup Experiments

Following
This article is more than 9 years old.

The following guest post is by Grace Ng, cofounder of Javelin.com, an enterprise software and services company for implementing Lean Startup. 

"We're launching our Minimum Viable Product in three months," entrepreneurs often tell me.

"Why three months?"

"We need to have these features in place to launch," the entrepreneur proceeds to rattle on about the list of features.

What is wrong with this situation?

By now, the concepts popularized by Eric Ries' The Lean Startup have become part of every day startup lingo. With that, the implications of each concept are misconstrued and loses meaning. MVP is used to describe the first version of a product launch, which it is not. Pivot is used loosely when teams abandon an idea on a whim in favor of another idea. Talking to customers is understood as asking your friends and peers what they think about your idea.

At Lean Startup Machine, I've worked with thousands of entrepreneurs to help break down their grand startup visions into quick experiments. I've seen people repeating the same mistakes when trying to apply Lean Startup principles to their ideas. Here are the most common mistakes that will help clear up any confusion.

1. You’re testing the wrong thing

All ideas are hypotheses that need to be tested. However, every time you have an idea, it's already a product or feature you want built; it's already a solution. Solutions that don't solve problems aren't solutions. Solutions without supporting data don't justify the risk it takes to proceed.

Testing the wrong aspect of your business can happen when you form the wrong hypothesis to test your idea. Forming the wrong hypothesis can happen when you misunderstand the problem and overlook the root cause.

For example, this is a common way of defining a hypothesis:

Shoppers want to see an activity feed of product purchases on their homepage.

This is what I call the "I have an idea!" way of forming a hypothesis. It is a solution-oriented problem hypothesis and will lead you into the tunnel vision of building too soon. Take a step back and ask, why?

Why?

Because…shoppers aren't buying enough products.

This is another common way of forming a hypothesis. But, this is the business perspective problem hypothesis. Startup founders see this as a huge problem they need to solve for, but it's a business problem, not the customer's problem. Ask why again.

Why?

Because customers don't think the product recommendations are authentic.

Aha! This is the correct hypothesis you want. One that is generated from customer interviews and defined from the customer's perspective. This is a hypothesis you can test with your customers and get feedback to see if you're going in the right direction. Defining your hypothesis from the customer's point of view helps you consciously create something that will deliver actual value.

Testing the wrong aspect of your business can also happen if you test the wrong underlying assumption of your idea. Each experiment you run should test your riskiest assumption; the assumption that is core to the viability of your hypothesis and most unknown, meaning you have little data or certainty as to the validity of the assumption holding true.

Teams often test assumptions that are not their riskiest and end up wasting time proving something they already know is true. By testing your riskiest assumption first you will surface the unknowns sooner, de-risk your idea faster, and speed up your progress.

At Lean Startup Machine, we've developed the Experiment Board, a free tool with templates you can use to construct effective hypotheses and quickly identify your riskiest assumption.

2. Your MVP is bloated

The most common misconception of the MVP is that it is a product that needs to be built. "How can I test with customers if I don't have anything built?" is the common pushback I get.

There are three simple ways you can actually "build" a MVP without writing a line of code! These are called exploration, pitch, and concierge experiments. Here's a video tutorial on how to conduct these experiments.

The purpose of the MVP is always to test your riskiest assumption and shorten the time it takes to get customer feedback. QuickMVP is a simple tool to quickly test if your idea is worth pursuing. You don't need to build all those features to test customer demand, unless it sets the baseline for testing your riskiest assumption.

3. Put a number on it, or I don’t believe you

"Let's just see what happens!"

While it is harder to quantify what success should look like without an initial baseline metric, you can still define success based on your business metrics, revenue goals, or cost of time and resources that you'll need to allocate. Writing down in advance what you expect to see holds you and your team accountable to what success looks like. Without defining this, it's easy to take any positive signals as validation and argue endlessly with your team about whether or not it's sufficient validation to move forward. At Lean Startup Machine, each team defines their minimum success criteria according to how big they perceive the problem to be for that customer segment.

4. You’re asking the wrong questions  

Talking to customers is often mistaken as go sell your solution to customers! But, it actually means, go interview your customers to discover their current behaviors and mental models, and listen for insights that will help you adjust your solution to meet their needs. Common mistakes when I see entrepreneurs interviewing their customers include: asking leading questions, selling too soon, talking too much, and not digging deeper when insights surface. Here are some tips for conducting effective customer interviews.

5. You dropped something in your pivot

Knowing when and how to pivot goes back to setting your success criteria. If the results from your experiment don't meet your success criteria, figure out why that is, change your hypothesis based on your learnings, and test again. This is called a pivot.

Common trip-ups entrepreneurs make include:

Throwing the baby out with the bathwater

Giving up on an idea without learning from the data. Sometimes, ideas for your next steps (insights from customers) are staring you right in the face! Remember to analyze your learnings and change your hypothesis accordingly, to pivot in a more informed and strategic way. In the Experiment Board, you can track your learnings and pivots over time to easily connect the dots and discover more ways to shape your idea into something customers want.

Validation doesn't mean full speed ahead

After pivoting to a solution that customers actually want, many entrepreneurs are tempted to step on the gas pedal and build out the grand vision. But the purpose of Lean Startup is to keep going through the build, measure, learn loop. Every new idea is a hypothesis that needs to be tested. Initial validation isn't permission to go full speed ahead.

The hardest thing about pivoting is getting everyone on the same page and looking at the same set of facts. That's why we created the Experiment Board. Teams who use it are able to stay aligned, argue less, and move faster.

It's exciting that more and more startups are running experiments to validate before building. Just watch out for these common mistakes and you'll be on your way to creating products that customers love, faster.