A conversion rate optimization (CRO) agency researches why visitors leave your site and tests ways to increase purchases or signups. Choose one that can explain how it identifies a problem and checks whether a change helped. A case-study percentage alone cannot tell you that.
The short version: Compare research, test output, evidence standards, design and development, and reporting. Ask who builds the changes and how the agency decides when a test has enough data. A high reported win rate can reflect weak stopping rules rather than better work.
The five questions below show what to ask and what a strong answer includes. The Remarkable runs a conversion optimization practice, so apply the same criteria to us.
How Do You Choose a CRO Agency?
Score every candidate on test velocity, statistical rigor, research depth, in-house production, and reporting focus. Those are the five criteria. Ask one direct question about each before committing to a second call.
Use the question column during the first sales call. Specific answers reveal more than a tool list or case-study headline.
Ask for one recent example after each answer. The agency should explain the decision, method, and result without exposing client secrets. Concrete operating details are harder to fake than polished principles.
| Criterion | Why it matters | Question to ask |
|---|---|---|
| Test velocity | Learning compounds with shipped experiments | How many tests did you ship last month? |
| Statistical rigor | Peeking at results invents wins that vanish | What threshold and sample size do you set first? |
| Research depth | Tests without research are guesses at scale | What research preceded your last hypothesis? |
| Design and dev | Tests stall when they wait on your engineers | Do you build the variants or do we? |
| Reporting focus | Conversion rate alone hides revenue per visitor | What number leads your monthly report? |
The rest of this post takes each criterion one at a time, with what a strong answer sounds like.
How Much Should a CRO Agency Test, and Does Statistical Rigor Matter?
Statistical rigor matters more than any other criterion. A disciplined agency sets the threshold and sample requirement before launch. It does not call a winner as soon as the dashboard looks positive.
VWO’s statistics roundup, citing Enterprise Apps Today, says 52.8% of CRO professionals lack a standard stopping point. Treat that figure as secondary reporting, not a fresh VWO study. Still, the question matters: ask how the agency decides when a test ends.
A strong answer names the method, threshold, sample requirement, and business-cycle rule. A weak answer is “when it looks clearly better.” Repeated peeking can increase false-positive risk. Sequential methods handle monitoring differently.
Ask what happens when traffic cannot support the planned test. A disciplined team may combine variants, extend the run, or choose another research method. It should not lower the standard after seeing an early result.
Once the stopping rules are clear, put test volume in context. Four tests may be strong for a complex product flow but weak for a high-traffic landing page. Compare output with the available traffic, engineering work, and required sample sizes.
More tests help only when rigor holds. Ask how many shipped last month and how many reached the required sample. High velocity without a stopping rule is fast guessing.
What Research Methods Should a CRO Agency Use?
A strong CRO agency studies both numbers and user behavior before writing a hypothesis. Funnel data shows where users leave, while interviews, surveys, and recordings help explain why. That research gives the test a reason to exist. Without it, A/B testing turns opinion into a larger experiment.
Ask how the team formed its latest hypothesis. “We thought the headline was weak” is a guess. A stronger answer connects observed behavior to a specific test.
Research takes time, and the scope should include it. A proposal that funds only test production may skip diagnosis. Check the promised research hours and methods.
Ask how the agency ranks hypotheses. A useful process considers expected impact, supporting evidence, effort, and traffic. The exact scoring model matters less than a visible reason for choosing one test over another.
Research continues after launch, as flat and losing tests provide evidence for the next hypothesis. The agency should record those lessons. Each experiment should build on what came before.
Want to see what disciplined CRO looks like?
Our conversion optimization engagement leads with research, sets significance thresholds before launch, and reports on revenue per visitor. Score us against this checklist yourself.
Book a Free Strategy CallShould a CRO Agency Bring Its Own Design and Development?
Usually, a CRO agency should bring its own designers and developers so approved experiments can move into production. Otherwise, tests depend on your engineering roadmap and can stall. The research only helps once someone can build the variants.
Ask who builds each test variant. If your team owns production, reserve capacity before the engagement starts. Otherwise, the first crowded product sprint can delay the testing plan.
Also ask who reviews quality before launch. The team should test analytics, devices, browsers, and important user states. A broken variant can waste the sample and create a false business conclusion.
A strategy-only agency can work when dedicated in-house designers and developers are available. Confirm their capacity and service-level expectations in writing. Do not assume spare production time exists.
How Should a CRO Agency Report Results?
A strong CRO agency reports on revenue per visitor alongside conversion rate. This matters once tests go live: a discount can lift checkout rate while reducing average order value. Revenue per visitor captures both effects.
Ask what leads the monthly report. If the answer is conversion rate, ask about revenue per visitor and order value. The reporting model should match your business economics.
For lead-generation businesses, revenue per visitor may arrive too late. Use qualified-lead rate, pipeline per visitor, or another downstream measure. The key is connecting experiments to value, not choosing one universal metric.
Watch how the agency frames wins. An unusually high win rate may reflect early stopping or low-risk ideas. Healthy programs include flat and losing tests, then record what each result changes.
How Do You Make the Final Call?
Score each candidate on all five criteria. Give statistical rigor and reporting the most weight. First, check whether you are at the right stage. When to hire a CRO agency covers traffic and readiness. Then compare pricing with what a CRO agency costs.
The right agency produces results that remain credible three months after launch.
New to the discipline itself? Start with what is conversion rate optimization.
We can help you think through the testing and production support your site needs. Share your traffic and testing history when you Book a Free Strategy Call. Use the five criteria above to assess whether our team fits.
Like this? Get the next one.
Short emails. New posts as they ship.
