A/B testing your pricing

By Steven Forth

A/B testing is the best practice for many parts of marketing these days. Most of the best SaaS companies use some form of A/B or multivariate testing in their marketing. Testing companies like Unbounce, in Vancouver BC, or Optimizely, in San Francisco, are growing rapidly. Web marketing automation platforms like Hubspot and Marketo include A/B testing modules, and popular content management systems like Wordpress have A/B testing plugins. 

In a recent conversation with Chris Herbert at SVPricing, a recruiting company in Silicon Valley, Chris commented that most people who are trained in pricing analysis are not trained in modern marketing approaches, and he called out A/B and multivariate testing as an explicit gap.

This opens the question, how should we be A/B testing our pricing?

One approach would be to test the price itself, and directly probe willingness to pay and price elasticity. This is problematic on many levels. In many jurisdictions, there will be legal issues around offering different buyers different prices based on an algorithm. Even if you can get away with this legally, assume that your buyers will discover this, usually at the worst time for you, and that there will be a public clamour about the lack of fairness and transparency of your pricing. In all likelihood, you will end up resetting everyone to the lowest price you tested and vowing never to do this again. A/B testing will not give you reliable insight into willingness to pay because willingness to pay is determined by many factors other than price.

Does this mean that A/B testing has no role in pricing? Not at all. 

There are three things you will want to A/B test: framing, messaging and packaging.

Framing

Framing effects are one of the obsessions of behavioral pricing. Basically, this is your expectation. If you think a bicycle should cost $200 (I was in Japan recently and this is the typical price for a standard bike there) then $700 sounds expensive. If you are a weekend warrior and expect to pay $2,000 for an entry level bike, then $700 for a bike leads you to think it is either a bargain, or more likely, that it won't meet your needs.

The order of presentation also has a big impact on framing. In the above example, I started with the less expensive example. This framing made the road bike seem relatively more expensive. Most of us would have had a different reaction if I had begun with the road bike and then given the example of the Mama Chari. Which is better will depend on the positioning one wants to establish in the marketplace. There is a restaurant chain in Vancouver called Cactus Club. In fact, it is quite expensive but it has positioned itself to appear to be mid market. There are other restaurants such as those run by the Top Table Group that are positioned up market. In fact, one can spend more at Cactus Club if one is not careful, and people often do. How you want to frame yourself relative to the alternatives is a critical part of your pricing strategy where framing has a big impact.

There is also internal framing. Many SaaS companies today have a three-tier structure of low, medium and high. Which offer should one present first and does it matter? The convention today is to present the lowest price first and thus frame using the lowest price (which makes the medium and high tiers seem more expensive). This is a great place to apply A/B testing. Four possibilities are shown below, vertical or horizontal presentation and low-to-high or high-to-low presentation.

Rather than defaulting to horizontal and low-to-high presentation, which is what almost everyone does, it is worth A/B testing (or even multivariate testing if you have enough traffic) different presentations to see how these impact distribution across the three tiers (the desired distribution is another topic that we will discuss in the future).

Messaging

Messaging is the most common thing that people A/B test. The critical thing to A/B test here are the value message and the buyer persona. The two should be closely aligned. Before you introduce a pricing tier, you should make it clear who the tier is for and the key value proposition. There is not a lot of space to do this, a couple of lines at most. The value messages introducing each tier are a critical thing to A/B test. This can get complicated as one wants to measure both (i) absolute conversions and (ii) distribution across the offers. Focus on one or the other depending on your current optimization goals. You will just confuse yourself if you try to optimize both at the same time. Most companies will want to optimize for conversions first and then work on the distribution.

Packaging

The main way to optimize for distribution across the tiers is by packaging (and pricing if necessary, but one does not want to be A/B testing the actual prices for the reasons given above).

Each tier will probably have a set of functionality attached to this. In most cases, this will take the form of basic for the lowest tier, everything in the lowest tier plus A, B, C for the middle tier, and everything but the kitchen sink for the highest tier. Getting the right set of functions in each tier, also referred to as packaging, is critical and can impact conversion and especially distribution. It is impossible to know what the optimum packaging is without A/B testing. The interactions are too subtle for most of us to work out. A/B testing is the best way to explore this. This will take a good underlying configuration management system, one that makes it easy to turn functions on and off, and a good user management system, one that makes it easy to track and compare different cohorts. It is hard to imagine how one would optimize packaging without A/B testing different configurations and tracking the results over time.

So is Chris Herbert right, is A/B testing part of the skill set of a pricing expert? I think the answer is yes. This will also feed into the long-term trend towards using more and more big data and machine learning in pricing. Over time, with enough A/B tests carried out in a systematic manner, machine learning may improve to the point that prediction is possible. The only way to get to this utopia is to begin a systematic A/B testing program of the factors that shape conversion rates and distributions.