# Creative test log: a completed hypothetical example

The Remarkable • September 29, 2026

**All names, creative concepts, dates, and results below are hypothetical. This is a teaching example, not a client case study or a performance benchmark.** The figures demonstrate a decision process; they do not establish statistical significance.

Blank planner: https://theremarkableagency.com/downloads/creative-test-planner.md
Guide: https://theremarkableagency.com/blog/creative-testing-framework-paid-social/

## Test question

- Test ID: DEMO_US_SOCIAL_RECONCILIATION_HOOK_01
- Product: an imaginary SaaS tool that helps operations teams reconcile spreadsheets.
- Owner: example media buyer. Reviewer: example creative strategist.
- Hypothesis: showing an unresolved spreadsheet row in the first 3 seconds will reduce qualified-activation CPA versus a product-screen opening.
- Primary metric: media spend / unique qualified activations.
- Qualified activation: a new user imports a spreadsheet and completes one reconciliation in the sample product.
- Diagnostic metric: outbound CTR, calculated as outbound clicks / impressions.
- Business guardrail: paid-customer media CAC must stay at or below $90 in this fictional plan.
- Minimum worthwhile improvement: at least 20% lower activation CPA than the concurrent control, with adequate evidence.

## Creative annotations

### A — control

Opening: a product screen highlights a matched row. Caption: “See which rows need a second look.”

The opening makes the product's function clear before the demonstration begins. Keep this as the comparison; do not edit it during the test.

### B — challenger

Opening: an unresolved row appears in a spreadsheet. Caption: “Which row is holding up your close?”

Only the opening 3 seconds change. The hypothesis is that a recognizable problem attracts more relevant users. The remaining demonstration, CTA, offer, duration, audience, and landing page stay the same.

Neither script claims a quantified time saving. A real campaign would need approved product evidence and licensed assets before production.

## Plan recorded before launch

- Delivery: September 1–7, 2026, using one account time zone.
- Measurement: a 7-day click attribution window; no view-through conversions in this example.
- Final read: September 15, after the last delivery day's attribution window closes.
- Method: planned platform A/B test with separate audience groups and a fixed budget.
- Budget: $600 per cell; $1,200 total media spend. Production costs are excluded.
- Planning assumption: $20 per qualified activation × 30 expected activations = $600 per cell.
- Average daily media budget: $1,200 / 7 = about $171.43 across both cells.
- Evidence rule: adopt only if the planned primary-metric test supports the improvement and the CAC guardrail passes.
- Budget rule: end at $1,200; do not add spend to chase a preferred result.

The expected 30 activations per cell is a budget assumption, not a sample-size calculation. This example has no underlying user-level dataset or confidence result. Its evidence requirement cannot be marked as met.

## Results at the final read

<table>
<thead>
<tr><th>Metric</th><th>A: control</th><th>B: challenger</th></tr>
</thead>
<tbody>
<tr><td>Media spend</td><td>$600</td><td>$600</td></tr>
<tr><td>Impressions</td><td>30,000</td><td>30,000</td></tr>
<tr><td>Outbound clicks</td><td>360</td><td>420</td></tr>
<tr><td>Outbound CTR</td><td>1.20%</td><td>1.40%</td></tr>
<tr><td>Qualified activations</td><td>24</td><td>21</td></tr>
<tr><td>Qualified-activation CPA</td><td>$25.00</td><td>$28.57</td></tr>
<tr><td>Attributed new paid customers</td><td>8</td><td>7</td></tr>
<tr><td>Paid-customer media CAC</td><td>$75.00</td><td>$85.71</td></tr>
</tbody>
</table>

Math: A's activation CPA is $600 / 24 = $25. B's is $600 / 21 = $28.57. B attracted more clicks, but its observed activation CPA was 14.3% higher. Neither difference is presented as statistically established.

The rows assume matching attribution rules and deduplicated events. Equal spend and impressions make the example easy to follow. They are not evidence of random assignment or an absence of bias in a real campaign.

## Decision

**Keep the control in place. Record the test as inconclusive; do not scale B.** The challenger did not meet the planned primary-metric improvement. The example also lacks the evidence needed to call either ad a reliable winner.

The CTR improvement alone does not justify a budget increase. Both variants stay below the fictional $90 media-CAC guardrail, but a guardrail pass cannot replace a primary-outcome win.

## Next brief

- Keep: the product demonstration, offer, landing page, and current control.
- Investigate: whether the problem-led opening attracted curiosity without enough intent to use the product.
- Next hypothesis: a more specific opening naming spreadsheet reconciliation may qualify attention better.
- Change: the opening caption only. Do not also change the offer or landing page.
- Before another launch: use the account's baseline conversion rate and a suitable test method to check the required sample and budget.
- What we did not learn: the reason for the observed difference, the best concept for every audience, or expected future CAC.

Do not transfer these numbers into a forecast. Transfer the habit: record the question, preserve the control, check the business outcome, and explain the next decision.