The short version: Choose a CRO agency on five criteria: test velocity, statistical rigor (fixed thresholds, no peeking), research depth (qual plus quant), whether they bring design and development, and reporting on revenue per visitor instead of conversion-rate vanity. Each one has a question that exposes the truth on the first call. Ignore the logo wall and headline win rate. A high win rate usually means the agency is peeking at results or testing only safe changes.
A logo wall tells you who signed a contract, not whose tests reached significance. The hard part of choosing a CRO agency is judging discipline you cannot see in a deck. This checklist makes it visible in one conversation.
Bias flag. The Remarkable runs a conversion optimization practice, so we are one of the agencies you might score with this list. The criteria are the same whether you hire us or a competitor, so use them on us too.
How Do You Choose a CRO Agency?
Score every candidate on five criteria and ask one question for each. Test velocity, statistical rigor, research depth, in-house design and development, and reporting focus. An agency that answers all five without reaching for a slide is worth a second call. One that pivots to its tool stack is telling you where its confidence ends.
The criteria sort the disciplined teams from the ones that A/B test button colors and hope. Use the question column live, on the first sales call, before they have your data.
| Criterion | Why it matters | Question to ask |
|---|---|---|
| Test velocity | Learning compounds with shipped experiments | How many tests did you ship last month? |
| Statistical rigor | Peeking at results invents wins that vanish | What threshold and sample size do you set first? |
| Research depth | Tests without research are guesses at scale | What research preceded your last hypothesis? |
| Design and dev | Tests stall when they wait on your engineers | Do you build the variants or do we? |
| Reporting focus | Conversion rate alone hides revenue per visitor | What number leads your monthly report? |
The rest of this post takes each criterion one at a time, with what a strong answer sounds like.
How Much Should a CRO Agency Test, and Does Statistical Rigor Matter?
Statistical rigor matters more than any other criterion. A disciplined agency sets its significance threshold and required sample size before a test launches, then waits for the sample to fill. Calling a winner the moment the numbers look good manufactures lifts that disappear in production.
This failure is common, not rare. About 52.8% of CRO professionals have no standardized stopping point for A/B tests, according to VWO. No fixed stopping rule means peeking, and peeking inflates false positives. Ask the agency how they decide when a test ends. A good answer names a threshold and a sample size. A bad answer is “when it looks clearly better.”
On velocity, more is better only if rigor holds. Ask how many tests shipped last month, then ask how many reached a pre-set sample. High velocity with no stopping rule is just fast guessing.
What Research Methods Should a CRO Agency Use?
A strong CRO agency runs both qualitative and quantitative research before it writes a hypothesis. Quantitative shows where users drop off. Qualitative shows why. Session recordings, user interviews, surveys, and funnel analytics together produce hypotheses worth testing. A/B testing alone, with no research, is opinion at scale.
The tell is in how they describe their last hypothesis. “We thought the headline was weak” is a guess. “Recordings showed users hesitating at the pricing step, so we tested clearer plan framing” is research. Ask what they looked at before they decided what to test.
This is also where thin engagements get exposed. Research takes hours that a cheap retainer cannot fund, so a too-cheap agency often skips straight to testing. You pay for tests that were never grounded in anything.
Want to see what disciplined CRO looks like?
Our conversion optimization engagement leads with research, sets significance thresholds before launch, and reports on revenue per visitor. Score us against this checklist yourself.
Book a Free Strategy CallShould a CRO Agency Bring Its Own Design and Development?
Yes, in almost every case. The single biggest cause of stalled testing programs is a queue of approved experiments waiting on your engineers. An agency that designs and builds its own variants ships on its own schedule. One that hands you tickets inherits your roadmap and your delays.
Ask directly: do you build the test variants, or do we? If the answer leans on your team, expect velocity to collapse the first time your product sprint fills up. The agencies that move fast own the full loop from hypothesis to live variant.
There is a narrow exception. If you have idle in-house design and development capacity dedicated to CRO, an agency that only supplies strategy can work. That is rare. Most teams that hire CRO help do so because they have no spare hands.
How Should a CRO Agency Report Results?
A strong CRO agency reports on revenue per visitor, not conversion rate alone. Conversion rate can climb while revenue falls when a discount lifts checkouts but drops average order value. Revenue per visitor catches that. It ties the program to the business.
Conversion rate as a headline is a vanity habit. Ask what leads their monthly report. If it is “we lifted conversion 8%,” ask what happened to revenue per visitor and order value. A team that already reports those is thinking about your business, not its slide.
Watch the framing of wins too. An agency that brags about a 90% win rate is usually peeking or testing only safe changes. Healthy programs produce plenty of flat and losing tests, because a losing test still tells you what your buyers ignore. Learning velocity beats a polished win-rate number.
How Do You Make the Final Call?
Score each candidate on all five criteria, weight statistical rigor and reporting highest, and pick the agency that answered live without reaching for a deck. Decide whether you are even at the right stage first: when to hire a CRO agency covers the traffic floor and the readiness signals. Then pressure-test pricing against what a CRO agency costs.
The agency that passes this checklist is the one whose tests will still look like wins three months after they ship.
Want to run this checklist on us in real time? Book a Free Strategy Call and ask all five questions. We will answer every one without a slide.
Like this? Get the next one.
Short emails. New posts as they ship.
