A Shopify wellness brand paid their performance marketing agency $8,500/month for 7 months. In month 1, ROAS moved from 2.2 to 2.7 — encouraging. By month 3, it was back to 2.4. Month 7 ended at 2.4. Seven months of agency fees, $59,500, and ROAS was 0.2 points above where they'd started. The founder knew something wasn't right by month 4. They waited until month 7 to act. Three metrics available in month 2 predicted the outcome. They just hadn't known to look for them.
The metrics agencies report vs the ones that actually matter
Most agency reporting is designed to look busy. Weekly decks with CTR, impressions, reach, engagement rate, CPM, frequency — numbers that show activity and platform-level efficiency. Very few of these predict whether you're building toward a profitable system.
The numbers that actually matter are three:
1. Landing page CVR from paid social (from Shopify UTM data, not the platform). This tells you whether the traffic the agency is buying converts into customers. You can have excellent CTR and terrible CVR — most brands do. Platform metrics look good; Shopify revenue doesn't move. If your agency isn't reporting landing page CVR from UTM-filtered Shopify data weekly, they're not tracking the most important number in the account.
2. Real ROAS — Shopify attributed, not platform attributed. The gap between Meta-reported and Shopify-attributed ROAS is typically 0.8–2.1 points. An agency reporting 4.2 platform ROAS while Shopify shows 2.6 is giving you a number that doesn't reflect what your business is actually making. The wellness brand's agency was reporting 3.1 ROAS throughout month 2-7. Shopify UTM data showed 2.1. Neither party had done the reconciliation.
3. Month-over-month creative output. How many new creative assets were tested last month? How many had statistically meaningful spend behind them? Agencies that stop testing creative within the first 90 days are coasting on the initial setup. Creative is the primary lever that moves ROAS over time. Stagnant creative libraries produce stagnant ROAS.
What a working agency relationship looks like in month 3
By month 3, a working engagement shows:
- Landing page CVR from paid traffic has moved at least 0.5 points from month 1 (or there's a documented reason it hasn't and a specific plan to fix it)
- At least 4–6 new creative assets have been tested, with documented hypotheses and results
- The agency has identified the primary constraint on ROAS — not a general "we're optimizing" statement, but a specific number that's the bottleneck (CVR is 2.1%, needs to be 3.0%; or creative is fatiguing at 6-week intervals; or audience size is too small for CBO)
- Shopify-attributed ROAS has moved by at least 10% from baseline
If month 3 doesn't show movement on landing page CVR and Shopify ROAS — not platform metrics, those two specifically — the engagement is coasting. That's the point where the conversation needs to happen.
What a failing engagement looks like (real patterns)
Four patterns that reliably predict an engagement that won't improve:
Reporting shifted to trailing averages. An agency losing ground on weekly numbers shifts to monthly averages to smooth the picture. "We're averaging 3.2 ROAS across the last 6 weeks" when the last 3 weeks were below break-even. Ask for week-by-week data.
Creative testing slowed after month 2. The initial enthusiasm produced 6 new creatives in the first 45 days. Month 3 onwards: one new creative per month, or fewer. The agency has settled into maintenance mode. Maintenance doesn't produce ROAS improvements.
Strategy calls became status updates. Monthly calls that used to include "here's what we're changing and why" became "here's what we did last month and here's what we're planning." Planning without diagnosis is not strategy.
No identification of the primary constraint. If you ask your agency "what's the single metric that, if improved, would most move our ROAS?" and the answer is vague or changes week to week — they don't have a diagnostic model. They're managing activity, not outcomes.
The wellness brand's engagement had all four by month 4. They stayed for 3 more months hoping for a turn. It didn't come.
The conversation to have before you make the decision
Before ending the engagement, one direct conversation:
"What is the primary constraint on ROAS right now, and what is the specific plan to address it in the next 4 weeks?"
A good agency gives you a specific answer: "Landing page CVR from paid social is 1.8%. It needs to be above 3.0% before we scale. Here's what we're changing on the landing page this week and how we'll measure it." A direct question deserves a direct answer.
If the answer is vague, ask once more: "Can you be more specific — what exact metric are you trying to move, from what number to what number, and in what timeframe?" If the second answer is also vague, you have your answer about the engagement.
The decision isn't about whether you like the team or whether they work hard. It's about whether there's a diagnostic model driving decisions or whether the account is being managed on vibes and optimism. The difference is visible in how they answer a direct question about the constraint they're solving.
If you're evaluating your current situation and want an independent read on whether the account is performing as it should — a Basic audit looks at Shopify-attributed ROAS, landing page CVR, and creative testing cadence and tells you where you actually stand. If you're also evaluating what to look for in a new agency, the 6 questions that filter bad-fit agencies give you the vetting framework.