BETA
This is a BETA experience. You may opt-out by clicking here

More From Forbes

Edit Story

Lean Startups Part 3: Most Changes Make Products Worse

Following
This article is more than 10 years old.

Lean startups are about innovating with a scientific mindset.

In Part 1, I discussed some of Eric Ries’s ideas, including the mental shift from “build, measure, learn” to “learn, measure, build”, the “magic test” and the two leaps of faith.

In Part 2, I reviewed finding the right idea to test, learning from pissed-off customers, the three engines of growth, and the difference between user testing and market research.

In this article (Part 3), I turn to Ries’s provocative declaration that “most changes make products worse.”

Innovation is often seen as adding valuable new features to a product. The underlying assumption is that what the innovator sees as improvements will be seen as improvements by the customer.

Ries disagrees:

In Lean Startup, we take those experiments directly into the heart of product development and we say, “Look, if you’re actually building a product and delivering it to customers, wouldn’t you like to know if the features you’re adding are actually making the product better or worse?” Most people know in their heart that the features they’re adding are making the product better, but I beg to differ. My experience working with hundreds of startups is that most changes make most products worse. So we’re trying to take those techniques and really bring them into our everyday work in building a product.

He gives a striking example from one firm where the three-step registration process of a website was redesigned.

The original registration process was an obvious opportunity for improvement, with poor page design and “crappy radio buttons”. So it was decided to improve it with big, bold, beautiful three-dimensional pictures. When a split test was done, the new improved screens did 20% worse than the old one, which was devastating for the design team.  What to do?

Ries explains:

The first impulse of most designers when that they get that kind of data is, “Well, it just looks so much better, let’s ship it anyway.”

I was like, “Are you kidding, ship it? No, no, no.”

The reason you don’t ship it anyway is not because we care so much about squeezing every last ounce of performance out of our registration process. It’s because when we fail, we have the opportunity to learn, if we take advantage of the opportunity.

Ries believed that the new registration design actually was better. But if it had a lower conversion rate, that meant that there was something about it that had been lost. There was something about the customers that they didn’t understand. So they decided to find out by doing split tests on each of the three pages of the registration process.

Ries says:

It was so simple it’s amazing that we didn’t see it. We found that the first two pages were awesome and had 20% improvement in the registration process, in getting people from page to page. But then we were giving up all those gains and by losing 40% of people on the third page, the download page, which was actually the simplest page. It was a page with this huge download button, with beautiful chrome and 3D specular highlights, just to make sure they didn’t miss it. Once we had that negative result, then we knew the right questions to ask.

We had been doing focus groups with the design all along. Our customers were always saying, “Yep, it’s great, it’s awesome. Love it, love it, love it.”

Once we knew there’s a download problem, our tests got smarter. We’d bring customers in and we’d watch them go through the process. instead of asking them, “Do you love this page?” we watched what they did.

We noticed this funny thing. We would be like, “Download the product,” and they would pause and they’d be looking around the page. We’re looking at our watches, like, “Dude. Okay. Click the download!”

Finally we would get frustrated, and we’d be like, “So why aren’t you clicking the download button?” The person would say, “’I can’t find the download button.”

We’re like, “Why not? What about this giant thing?”

It was so big and obvious that users couldn’t see it!

When we fixed it, all the numbers went up. But the insight was more than just best size and color for the download button. The insight was, “Oh, we think about how we use our computer different than this person, and we need to be aware of that in our design.” So that was actually a very important moment there.

Work isn’t done until it is validated by the customer

The definition of what it means to say that work is “done” is a matter of intense discussion in the Scrum and Agile world, as reflected in an illuminating article by Mayank Gupta. A common definition would include completed code, unit tests completed, integration tests completed, performance tested and approved by the client or the customer proxy (the product owner ).

What Ries is suggesting is that it’s not enough that the client or customer proxy has approved the work in terms of agreeing that the feature conforms to what was asked for. In addition, you need to do a scientific experiment to determine whether the feature actually works better with real users in the real world. Is the product really better as a result?

Stop adding features & get rid of the crappy stuff

Continually adding features may in fact lead to a customer–driven death spiral, in which the product becomes increasingly unusable by the people in the real world. Although the product may in some objective or technical sense be getting better with more capabilities, it is actually getting worse in terms of the customer experience.

This underlines the need for a shift in focus from pushing more stuff at customers (inside-out focus) to an outside-in perspective of understanding the customers world and making the experience better for them.

A big part of Steve Jobs’ success at Apple [AAPL] has been in shedding ideas that don’t make the final cut and simplifying products so that they are elegant and easy to use. Carmine Gallo in an interesting article, “Steve Jobs: Get Rid Of the Crappy Stuff” writes:

“Editing also leads to great product designs and effective communications. According to Steve Jobs, ‘People think focus means saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully. I’m actually as proud of the things we haven’t done as the things I have done. Innovation is saying ‘no’ to 1,000 things.’”

Imagine how much better our lives would be if all firms “got rid of the crappy stuff”.

What Eric Ries is doing for us in lean startups is giving us a methodology for accomplishing this on a systematic, scientific basis.

In this series of articles on Lean Startups:

Part 1: Something New In Innovation: Lean Startups

Part 2: A Scientific Method For Creating Innovation

Today: Part 3:  Most changes Make The Product Worse

Coming tomorrow: Part 4: Implementing Lean Startups: Kanban vs Scrum

______________________

Steve Denning’s most recent book is: The Leader’s Guide to Radical Management: Reinventing the Workplace For the 21st Century (Jossey-Bass, 2010).

Follow Steve Denning on Twitter @stevedenning

Join the Zurich Gathering For C-Suite Leaders with Steve Denning Zurich Sep 12, 2011