You get what you measure, so running tests without a clear plan is just creating noise. The most successful operators treat testing not as a tactic for a quick lift, but as a systematic process for learning. It all starts with defining your hypothesis and the specific metrics you'll use to measure success. This discipline is what separates meaningful, compounding insights from a series of random guesses.
How do I form a good hypothesis?
A strong hypothesis isn't a vague guess like "a new hero image will increase sales." It's a specific, testable statement based on data or user research. As Oliver Palmer explained on The eCom Ops Podcast, you need to practice Hypothesis-driven A/B testing. This means digging into your analytics, session recordings, or customer feedback to find a problem. Your hypothesis should clearly state what you're changing, who you expect it to affect, and what outcome you predict.
Shahram Anver made a great point on Honest Ecommerce about always starting with "why." Instead of asking "what should I test?" first ask "why are we doing this?" For instance, if you see a big drop-off on a product page, your "why" might be that customers don't have enough information to make a decision. Your hypothesis could then be: "By adding a 'How It Works' video to the product page, we will reduce bounce rate and increase 'Add to Cart' clicks because we are addressing user uncertainty."
What metrics should I focus on besides conversion rate?
While conversion rate is important, it doesn’t tell the whole story. As Chloe Thomas lays out on the eCommerce MasterPlan podcast, you need to think about a range of goals: financial, customer-focused, and operational. Your testing strategy should reflect this. If your business goal is to increase profitability, you should be testing things that could improve your Average Order Value (AOV) or Customer Lifetime Value (CLV).
When planning a test, define a primary metric, but also identify key counter-metrics. For example, you might test a promotional banner that boosts conversion rate (your primary metric), but you need to watch AOV to make sure it isn't just driving sales of low-margin items and hurting overall profitability. Tim Wilson, speaking on Honest Ecommerce, emphasized the need for defining what success looks like before launching anything. This means agreeing with your team on the specific KPI that determines the winner before the test even starts.
How do I prioritize what to test?
Your resources are finite, so prioritization is everything. Andrew Faris makes the case on his podcast for having a real process for it. A key part of that process is focusing on the biggest questions and the parts of the funnel closest to the checkout first, as those are most likely to have the largest impact on revenue. Don't spend a month testing the color of a button on your 'About Us' page; focus on the product page, cart, and checkout flow.
Ezra Firestone echoed this on The eCommerceFuel Podcast, noting that you should prioritize tests based on their potential impact and your confidence in the hypothesis. If you have strong data suggesting a major point of friction in your checkout, that's a high-priority test. This structured approach is central to making Data-Driven Decisions, ensuring you're always working on the most valuable problems first.
Ultimately, a rigorous testing program is about building a deep, empirical understanding of your customers. It moves you away from relying on so-called "best practices" that might not even apply to your audience, a point Kurt Elster and Paul Rita drive home on The Unofficial Shopify Podcast. By forming clear hypotheses and measuring what matters, you turn testing from a guessing game into a reliable engine for sustainable growth.