How To Use Customer Feedback: Priority and Roadmaps

For how crucial customer feedback is to everything we do in the subscription business, there is still a divide between customer feedback and the value that you can draw from it.

Bridging the gap calls for tying feedback back to the job your customers have hired your product to achieve on the one hand; and working the right solutions into your product roadmap, on the other.

The tension between the past and the new

Towards a working definition of what the ‘right’ solutions are, Part I of this series on customer feedback will guide you through solving for the problem rather than the pain.

This post, Part II, is focused on the other side of the challenge: how to balance Customer (feedback) with Product, acknowledging that the needs of your present customers can either be in sync with or at odds with the natural progression that your product has within a market niche.

The right solutions are right because, and only if, they make sense for both Customer and Product.

What does it mean for customers (and consequently, their feedback) to be at odds with Product, though?

Sachin Rekhi, who was product manager at LinkedIn for close to four years, explains, “the value of the feedback is […] the synthesis of the learnings from that feedback. For example, you might notice that a subset of customers are asking for very specific power-user functionality and you might ultimately decide that rather than addressing the feedback, the user segment is no longer a target customer. Or you might hear a lot of feature suggestions / confusion / issues with a specific feature and instead of simply addressing the specific feedback, you realize you need to re-think the entire experience from scratch to avoid the current user sentiment.”

Here’s another way of looking at the challenge.

Consider these propositions:

• Every solution you implement is a change to your product, and not all solutions are equal when you consider how you want your product to evolve.
• If a solution will fix a problem your customers are having, then it represents a change to your product that’s worth considering. Even if it’s a small number of customers.

Each proposition seems to make perfect sense taken in isolation from the other. Taken together, however, they lend themselves to, as French philosopher Jacques Derrida like to refer to it, the tension between past and new.

The right solutions are ones that work for the now

Recognizing this tension between past and new turns our initial question into one of prioritization.

Sachin, again, this is a brief excerpt from his blog that shines a light on how he thinks about balancing features in his product roadmap, “I think about my product’s roadmap as a modern portfolio, where I’m constantly balancing risk/reward across a diverse portfolio of product initiatives […] constantly balancing bugs, performance/scale issues, poor performing functionality, customer feedback, brand new features, and entirely new products.”

The right solutions are the ones that work for the now, however you want to define it – the next quarter, the next six months; they are solutions that that will both cater to your present customers in the best way possible (depending on your goals) while pushing your growth within the market a little further over an immediate period of time.

Some of the best product managers out there have taken a shot at coming up with systems and processes that balance feedback with effort using prioritization as a tool.

Ryan Singer’s got you covered if you’re looking to prioritize features for a product you haven’t started building as yet.

Martin Eriksson’s got you covered if you think the prioritization process you have in place doesn’t quite match the size of your business (the bigger the business, the more costly it is to take risks – this affects how you prioritize features).

The problem (with them and the other models I’ve researched – Jim Semick’s benefit vs cost model, Andrea Saez’s theme based model, caroli.org has a great list of more) is that none differentiate between sources of feedback and manage to account for the tension between the past (your existing customers) and the new (your growth or your future customers).

The RICE Model

Consider Sean McBride’s model of prioritization – the RICE model, which he implemented at Intercom.

The RICE model is a method of prioritizing features that balances three parameters of a feature you’re considering (reach, impact, and confidence) against the effort that it takes to build it. It’s sleek, intuitive, and incredibly popular.

Here’s a breakdown of how it works and how you can use it to account for the tension between past and new.

The parameters within the model

Reach defines the number of people your feature will affect over a given period of time (say, a quarter or six months). Reach is driven by numbers, customer and market metrics.

Impact, complementing Reach, defines how powerful the affection will be for an individual. This helps visualize whether a feature will affect a large number of customers in a small way, for example, or a small number of customers in a big way. Reach is driven by intuition and scored on a scale of zero to three (for increasing impact – .25 would be minimal impact, for example, whereas 3 would be massive impact).

Confidence acts as a lever against Reach and Impact, allowing you to factor how you feel about your numbers into your decisions. If data supports your Reach and Impact scores, you can mark your confidence down as 90%, for example, lower if it doesn’t. As McBride says, ‘If you think a project could have huge impact but don’t have data to back it up, confidence lets you control that’.

The formula

Reach * Impact * Confidence / Effort = RICE score. (where effort is scored like Impact on a scale of 0 to 3)

The higher the RICE score, the better.

Where RICE excels

• RICE quantifies parameters that are usually driven by intuition with hard data, it even factors in how confident you are about the data.
• RICE helps compare projects and features at a glance.
• It allows space for improvisation in a roadmap (if there is a need for it), working more as a rule of thumb than a rule.

Where RICE falls short

Reach, Impact, and Confidence can each be quantified differently for the customers you have and the customers you want. While it works for a team focused on onboarding, Rice doesn’t work for a team trying to balance the past with the new.

Using RICE to implement better value from feedback

To figure out how the RICE model could be leveraged better, I turned to Chargebee’s product managers, asking how RICE could be used to account for the tension between past and new.

The answer we arrived at was simple: catalog your data so you can compare how a feature affects your present customers with how it will affect your presence and growth with the market. Prioritize for the segment your immediate goals are targeting. If your goals are to grow within a new market, you would prioritize features that help you grow. If your goals are to reduce churn, on the other hand, you would prioritize features that help your existing customers find more value in your product.

In other words, a) compare the RICE score of a feature when calculated using metrics that are indicative of the needs of your present customers vs the RICE of a feature calculated using metrics that are indicative of what your product needs to grow and b) let your goals do the rest.

Step 1: Stop putting all your feedback together. Separate the feedback that informs on present customers from the feedback that informs on market trends.

Internalize that your customer feedback is coming from different places, from different perspectives, and should be weighed differently.

Sachin Rekhi has a great perspective on how to catalog your sources of feedback.

The heart of my criticism of RICE is that a score you calculate using one set of feedback will look very different from the score you calculate using the other.

Here are a few examples of sources that are indicative of the needs of your present customers:

• Customer interviews
• Customer surveys
• Feedback from support
• KPIs on a feature

On the other hand, here are a few examples of sources that indicate what features you need in order to grow:

• Prospect calls
• Competitor features
• Social media, the product ecosystem that your customers are using your product in conjunction with, and market share research (for data on trends within the market)

Step 2: Calculate separate RICE scores for a feature using the two sets of metrics.

Reach would translate to a) the reach within your existing customer base and b) the reach within your potential customer base, respectively. Impact, similarly, would translate to a) the impact that the feature would have on an existing customer and b) the impact the feature would have on a new customer, and so on. (Note that your effort score would stay the same in both cases, though)

Step 3: Compare the scores (RICE^e for existing customers, RICE^f for future ones) for an idea of how to prioritize for your goals.

Here are a few examples of features we’ve pushed out in the last three months, wrestling with the tension of past and new.

RICE^e = RICE^f (prioritized highest)

Our email notification revamp (in beta at the moment) falls in this category, hands down. The effort it called for from product, design, and sales was huge, but it was a feature that we could tie to requests and jobs on both sides of customers, present and potential. Email communication is a crucial part of billing and all of our customers were using emails – either their own notifications, a third party’s or version one of ours. The reach and impact that a revamp would have on both churn and growth made this feature a priority despite the effort, with the almost identical RICE scores for both sets of data points.

It’s easy to imagine the other features that might fall into this category – the ones that ensure you meet basic requirements, industry standards, and a critical set of use cases. Features that everyone who’s using your product will need.

RICE^e < RICE^f (prioritized to meet growth goals)

The File Attachment feature, which lets you attach important files to subscriptions so that they’re permanently on record, fits this bill. The effort it called for was comparatively low, and data from our sales team and market research indicated it was important to bigger businesses that demanded an easily accessible paper trail. The RICE score for File Attachment was tipped towards our potential customers rather than our present ones, but we prioritized it for this quarter because it helped us meet our growth goals.

RICE^e > RICE^f (prioritized to meet churn goals)

The Slack integration lies on the other side of the coin. Tied to the job of having subscription info handy when talking to teams or teammates, the Slack integration was heavily tipped towards our present customers, with a record adoption rate no more than a few hours after it was released. A perfect example of how you ought to prioritize with your immediate goals in mind – Slack was a customer request that had to be balanced against the needs of the product if it was going to bring maximum value to Chargebee as a whole.

Maximum value is a question of priority; priority is a question of goals

If you’re directing the ‘What should I prioritize first?’ question to a seasoned entrepreneur, the answer you’ll likely receive is ‘well, what do you want to do?’

In essence, what I hope you take away, whether you want to try the RICE model on for size or not, is the idea that prioritization looks very different when informed by data from your existing customers and data from your prospects and the market.

This means that sitting under the challenge of prioritizing features is the question of prioritizing data, because you don’t have the resources to build everything you want to, you never will.

It might be worth cataloging and segmenting your feedback so your roadmap caters to customers in a way that aligns with your immediate goals.

Between tying feedback to the job that your customers have hired your product to do and balancing those jobs against those you want your product to accomplish, you’re all set to wring the most value out of customer feedback.

Immense thanks to @Prasanth Kothari, one of Chargebee’s ever-inspiring product managers, for helping me put this piece together.

What model are you using to prioritize features? Do you agree with the proposition that cataloging and segmenting your feedback will help prioritize in a way that works for your goals? Have you ever used RICE? I want to hear from you, my email ID is yohann@chargebee.com