Open almost any Google Ads account and you'll find the same thing: a handful of RSAs, each with a handful of headlines, and Google's Ad Strength meter quietly deciding which combination gets called 'Best.' Nobody ran a test. Nobody checked a sample size. Google's automated asset rotation just noticed one combination pulled ahead early, gave it more impressions, and let that early lead compound into a self-fulfilling label. That's not a test. That's a coin that landed heads twice and got promoted to lucky.
Why Google's Own Optimization Isn't a Real Test
Google's ad rotation is designed to maximize short-term performance, not to answer a question. It weights impressions toward whatever's currently converting best, which means the variant that got a strong first week — often for reasons that have nothing to do with the copy, like a temporary dip in competitor impression share or a burst of branded traffic — starts snowballing. More impressions means more data means more perceived confidence, even though the underlying sample per headline combination is still thin. By the time you glance at 'Best' vs. 'Low,' you're not looking at a controlled experiment. You're looking at whichever asset got a head start.
The Sample Size Threshold I Actually Use
Before I'll call any headline, description, or CTA a genuine winner, I need enough conversions per variant to trust the number, not just enough clicks. Clicks tell you about attention. Conversions tell you about revenue. I generally won't draw a conclusion on an asset until it has accumulated at least 100 conversions on its own, or the test has run long enough to cover multiple full business cycles (at least 2-3 weeks, longer for anything with weekly buying patterns like B2B or high-ticket ecommerce). Under that threshold, I treat every result as noise, regardless of what the Ad Strength meter says.
- Minimum ~100 conversions per asset combination before treating a difference as real, not directional
- Minimum 2-3 full weeks of runtime to smooth out day-of-week and pay-cycle effects
- A 95% confidence threshold on the conversion rate difference before retiring a variant
- Separate thresholds for high-ticket/low-volume accounts, where I extend the runtime rather than lower the bar
That last point matters more than most advertisers admit. A lead-gen account generating eight conversions a week can't hit statistical significance in two weeks no matter how badly the dashboard wants to declare a winner. In that case, I stretch the test window rather than accept a weak conclusion — a slower, honest answer beats a fast, wrong one.
Isolating the Variable You're Actually Testing
The bigger problem with most RSA 'tests' isn't sample size — it's that nobody isolated a variable in the first place. RSAs combine multiple headlines and descriptions automatically, so if you swap three headlines and a CTA at the same time, you have no way of knowing which change moved the needle. I run tests one variable at a time: pin the headlines you're not testing into fixed positions, change only the one element in question, and let the rest of the ad stay identical to the control. It's slower than letting Google mix and match everything, but it's the only way to attribute a lift to a specific piece of copy instead of an unexplainable blend.
An ad that lucked into more impression share isn't a winner. It's just louder.
Once the sample size is there, I pull the asset-level 'combinations' report rather than trusting the summary Ad Strength label, and I check the result against a basic significance calculation on conversion rate, not just CTR. High CTR with a mediocre conversion rate is a headline that's good at getting attention and bad at getting the right people to click — which is exactly the kind of signal that fools both Google's optimizer and a rushed human reviewer. I've written before about how junk conversions distort Smart Bidding; the same distortion applies here. An RSA variant that wins on volume but loses on lead quality isn't a winner, it's a trap, which is why this review has to happen alongside a clean read on conversion tracking, not in isolation from it.
- ✓︎Isolate one variable per test — pin the rest, change only the headline/CTA in question
- ✓︎Wait for ~100 conversions per asset combination, not just clicks or impressions
- ✓︎Extend the test window for low-volume accounts instead of calling it early
- ✓︎Pull the asset combinations report and check conversion rate, not just CTR or Ad Strength
- ✓︎Confirm the winning variant's conversions are real leads, not junk, before retiring the loser
- How long should an RSA test run before I trust the result?
- Long enough to hit roughly 100 conversions on each asset being compared, and at least 2-3 full weeks to smooth out weekly buying cycles. For low-volume accounts, extend the timeline rather than accepting a smaller sample.
- Should I trust Google's Ad Strength or 'Best' label at all?
- Treat it as a rough directional signal at best. It reflects impression-weighted performance under Google's own optimization logic, not a controlled test with a defined sample size or isolated variable.
- Can I test more than one headline at a time?
- You can, but you lose the ability to attribute results to a specific change. Pinning positions and changing one element at a time is slower but is the only way to know what actually caused the lift.