CRO · By Arav Sahni · FutureSource

A/B Testing Without Fooling Yourself

A/B testing is supposed to remove opinion from decisions. Done badly, it does the opposite — it dresses up a guess in the costume of data and gives a team false confidence to roll out something that does not actually work. The fix is discipline, not more software.

Test One Big Thing

Changing five small things at once tells you nothing about which one moved the needle, and bold changes produce clearer, more reliable results than timid ones. Test one substantial change at a time — a different headline, offer, or page structure — so the outcome is unambiguous and worth shipping.

Reach Real Significance

Calling a winner after thirty conversions is guessing, not testing. Decide the required sample size and run duration before you start, let the test cover at least one full business cycle, and resist the strong urge to stop early just because an early lead looks exciting.

• Set the sample size and end date before launching.
• Run at least one full week to cover weekday and weekend behaviour.
• Do not peek and stop the moment a variant looks ahead.

Watch the Metric That Pays

A variant that lifts clicks but lowers revenue is a loss disguised as a win. Always tie the test to the metric that actually pays you — qualified leads or revenue — rather than a vanity number higher in the funnel that feels good but does not reach the bank.

Keep a Test Log

Record every test, its hypothesis, and its result, including the losers. Over time that log becomes your most valuable asset: it stops the team from re-running settled questions and turns scattered experiments into a compounding understanding of what your specific audience responds to.

Written by Arav Sahni, FutureSource — Montreal. Book a strategy call.