Why Your Organization Needs An Efficient Experimentation Engine

With the rising prevalence and accessibility of the cloud, developing products has become increasingly viable. And as many products compete against each other, it is critical to build products that matter to customers. Hence, experimentation becomes the most valuable weapon in a product manager’s arsenal to stand out in the highly competitive SaaS market.

Some of the most successful companies today, like Salesforce, Shopify, Zoom, Intuit, Google, and Amazon, are also – not surprisingly – the most innovative companies. And the cornerstone to their innovation is experimentation. These companies conduct thousands of experiments annually. Because while they discard more ideas than they develop, they are confident of a few big wins that make their experimentation worthwhile. Let’s delve further into why product experimentation is necessary and how to build an efficient experimentation engine in your organization.

Why You Should Experiment

Firstly, experimentation is necessary for invention and to emerge as category creators or leaders. When it comes to workplace innovation, ServiceNow is one of the leading SaaS companies with an impressive market capitalization of $127.9B. Their product digitizes workflows within a company to make day-to-day operational processes more efficient. Ranked as the world’s most innovative company by Forbes in 2018, ServiceNow’s software platform automates tasks across functions to meet the needs of employees. Driving the digital transformation of organizations, ServiceNow beat out legacy IT service-management software players like BMC Software, Hewlett Packard Enterprise and CA Technologies to claim more than half of the IT service management market. 

Experimentation aids customer-centricity (through customer-driven innovation) and creates sustainable competitive advantages for companies. Adobe, for instance, made several successful strategic decisions to remain competitive and relevant to customers. The key to their success was that they never stopped looking for new ways to wow their customers. Adobe sold software in a box before moving to the cloud and a subscription model with their Adobe Creative Cloud offering in 2013. Since its transition, Adobe has seen a 3x increase in revenue and a 17x increase in market cap. Slack, for example, changed the way people communicated and collaborated thanks to the convenience it provides, as a comprehensive platform with over 900,000 integrations that ensure ease of usage.

Experimentation is also vital for hyper-growth. Back in 2000, Amazon was an e-commerce company struggling with the complexity of scaling. Amazon established some solid internal systems to deal with the hyper-growth it was experiencing, which became the foundation for AWS. AWS began its journey to enable third-party merchants like Target or Marks & Spencer to build online shopping sites on Amazon’s e-commerce engine. Today it allows any company or developer to run their applications on their cloud infrastructure platform. It has since grown to become one of the fastest-growing B2B businesses in history.

Best Practices

Now that we’ve covered the “why” of experimentation, let’s look at a few pointers that will help you experiment successfully: 

Run a lot of experiments: Jeff Bezos claims, “Our success at Amazon is a function of how many experiments we run per year, per month, per week, per day.” The company runs close to 2k experiments a year, and this comes in handy because:

Most experiments fail: At Uber, maybe 20–30% of their experiments work. Less than 50% of the experiments conducted at Amazon, Microsoft, and other innovation-led companies provided the desired results. Bezos explains, “Given a ten percent chance of a 100 times payoff, you should take that bet every time. But you’re still going to be wrong nine times out of ten. We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs”. 

Moreover, if you do not fail, you do not learn, and hence, you do not improve. After all, experimentation is nothing but validated learning. You’ll understand why further in the article when we discuss the experimentation cycle.

Testing, testing, and more testing: Netflix explains their experimentation platform is all about A/B testing in their blog. Everything is thoroughly tested, from deciding which images to associate with media titles (that resulted in 20-30% more viewing!) to the complete redesign of their UI layout.

Be data-driven: As much as prototyping and testing are essential to creating a live version of your idea, how do you determine if it’s a success? By selecting the right metrics for success. For example, when we (Chargebee) released the second version of our product catalog, the metric we measured to define success was the number of migrations customers made from the first version.

Know when to shut it down: While experimentation is a must, ensure that it doesn’t come at the expense of product stability and time to market. Often, getting a product out is more important than worrying about finesse.

How to Effectively Experiment

Before we discuss the experimentation cycle, here are some popular experimentation methodologies that are often followed:

Outcome-Driven Innovation: It’s an innovation process that is centered around understanding how customers measure value. Once companies understand that, they can align marketing, development, and R&D with these metrics and systematically create customer value. ODI has an 86% success rate, which is five times higher than other experimentation methodologies.

The “Lean” Methodology: The lean method focuses on rapidly transforming new product or service ideas into iterative experiments. It entails creating prototypes known as Minimum Viable Products (MVPs), moving quickly to gather feedback from constituents on these MVPs, and then developing iterations of their MVPs based on that feedback. “The Minimum Viable Product (MVP) is that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort,” says Eric Ries of The Lean Startup. A true lean success, Airbnb paid professional photographers to visit hosts and make listings more appealing in one of their most well-known experiments. Customers responded, and listings with professional photos outperformed listings without professional photos by 2.5 times.

Regardless of the experimentation methodologies followed, the basics of an experimentation cycle remain similar for most companies. For a better idea, here is how we run experimentation at Chargebee:

The first step involves ideation, or to be more specific – figuring out what features to release. It depends on our business goals for the year, playing by our strengths (penetrating markets in which we have a strong presence), and most importantly – what our customers want. Most of the features we release are new, whereas the rest are feature enhancements or technical improvements. Or, in some cases, the sole purpose of experimentation could also be to gather insights on new segments/verticals.

Once we decide on the features, step two involves asking ourselves, “How do we solve the customers’ problem?”. The first component involves internal introspection, where our product managers, designers, and architects get together to discuss the problem, solution, deadlines, and customer needs at length. The second component is external experimentation, where we develop screens MVPs – a simple product version that is viable to test. Here, companies can also conduct A/B testing on the MVP to refine the feature functionalities further.

We push the MVP through an Early Adopter Program (EAP) in the third and most crucial step. The EAP enables us to validate our experiment through our customers apart from helping us build better relationships with them. It also gives our customers a chance to preview features and test how it syncs with their business workflows. Our ideal EAP audience includes:

• Customers who can benefit from our MVP.
• Customers who have asked us for the feature.
• Customers who have been with us for a long time.

We have a varied customer base since we support subscription businesses across various industries, from spa services, coffee chains to automobile companies. Once we narrow our audience, we enable the feature/s (MVP) in their Chargebee test sites. We then gauge if our future product feature has the necessary customer validation – did we solve the customers’ problem? For instance, we pushed the second version of our product catalog via EAP. Once the version was validated and accepted, we enabled the feature and based on feedback, we are continuously working on improving our product catalog, whether it’s bug fixes or coming up with ancillary features.

The final step is launching the product. Here are a few questions we always ask ourselves before launching:

• What is the validation from the market (product-market fit / feature-market fit / product-portfolio fit/ product usage)?
• What are the insights that we have gained from the EAP and how to drive feature adoption?
• What are the product metrics for feature adoption success?

Building a Culture of Experimentation

Experimentation is not a one-time affair. Besides being scientific – finding out what customers want and continually testing, experimentation is also a mindset – constantly putting wild ideas, new business models, new products, and new processes to the test. Hence, you must build experimentation into your organization’s culture from day one.

For instance, at Chargebee, we have taken the initiative of celebrating free-thinkers, experimenters, and innovators alike in our Champions of Change Summit. Our keynote speakers included Patty McCord (former CTO of Netflix) and Bill Macaitis (former CMO and CRO at Slack, Zendesk, and Salesforce). 

It’s always easy to get comfortable with being “good enough.” But to stand out and be successful, the key is to foster growth, change, and experimentation actively.