A Shopify wellness brand hired their last agency based on a deck showing 4.7 ROAS case studies. 4 months in, ROAS sat at 2.4 and the account looked identical to day one. Creative tested: 3 variations. Landing page changes: zero. When they asked the agency to explain, the response was "we're still in the learning phase." The case studies were from a different category, different margins, different traffic mix. The deck was real. The relevance was not.
Why Most Agency Vetting Processes Fail
The typical vetting process evaluates what agencies want you to evaluate: case studies, client logos, platform certifications, team size. None of these tell you whether this agency will work for your specific account.
The case study shows a supplement brand at 4.6 ROAS. You sell fashion. The supplement brand's margin structure, conversion window, and creative strategy share nothing with yours. The ROAS number isn't dishonest — it's just irrelevant.
The right vetting process evaluates execution quality, communication speed, creative output, and how the agency diagnoses a problem they don't yet understand. That requires asking different questions.
6 Questions That Filter Bad Fits
1. "Walk me through the last account you took over where ROAS was flat for more than 2 months. What did you find and what changed first?"
Listen for: a specific diagnosis (not "we tested more creatives"), a clear first action, and a timeline. A good answer names the actual constraint — landing page, creative fatigue, audience overlap, offer clarity — and explains why that was first.
A red flag: "We rebuilt the campaign structure and tested new audiences." That's activity, not diagnosis.
2. "What does month 1 deliverable list look like for an account at my revenue stage?"
A good answer includes: a full account audit (ad account, Shopify Analytics, landing pages, creative library), a break-even ROAS calculation based on your margins, and a list of changes made vs changes queued for testing.
A red flag: "We'll start with learning campaigns and see what the data tells us." That's 30 days of spend with no framework.
3. "How do you measure success — what metrics do you report weekly?"
A good answer includes Shopify-reported ROAS or revenue (not just platform ROAS), CVR from paid traffic (from Shopify Analytics, not the ad platform), and new vs returning customer rate.
A red flag: platform ROAS only. An agency that doesn't reconcile platform data with Shopify data is optimising a number that's 12–22% inflated.
4. "How many creative variants do you typically test per month for an account at my budget?"
At $10K–$20K/month ad spend, a productive agency is testing 4–6 new concepts per month — not just resizing existing assets. The answer should include a clear process: how briefs are written, what approval looks like, how performance signals are read.
A red flag: "We test what performs well and iterate." That's circular — it doesn't describe a testing framework.
5. "Can you show me a landing page recommendation you made that changed CVR?"
A good agency will have a specific example: the original page, the change made, and the CVR result. If the agency has never touched a landing page and only operates in the ad account, they're solving half the funnel.
A red flag: "We focus on the ad side — landing pages are your team's responsibility." A broken landing page makes good ads worthless. An agency that won't engage with it is working with one hand.
6. "What's your process when something isn't working in month 2?"
A good answer describes an escalation protocol: what the diagnosis cadence looks like, when strategy changes vs incremental iteration, and what triggers a full audit vs a creative refresh.
A red flag: "We continue to optimise and trust the process." By month 2, an agency should know whether the constraint is creative, offer, landing page, or audience — and be able to explain which and why.
Red Flags in Proposals and Calls
- Case studies from a different product category or margin structure with no acknowledgment
- Guaranteed ROAS numbers (no one can guarantee an outcome in a live auction system)
- No mention of Shopify Analytics, CVR, or landing pages
- Proposal describes "strategy development" in month 1 (you're paying for execution, not discovery)
- Long-term contracts with no performance-based out clause
What Month 1 Should Look Like
A working engagement produces these in the first 30 days. For a full month-by-month breakdown of what a 90-day agency engagement should deliver, see the 90-day performance marketing roadmap.
- Full account audit: ad structure, creative history, ROAS by campaign, CVR by source
- Break-even ROAS calculated from your actual margins
- At least 2 new creative concepts in testing
- One change to landing page or offer based on audit findings
- A weekly reporting template that includes Shopify revenue alongside platform ROAS
If your current agency or a prospect can't describe month 1 in those terms, you're in discovery, not execution.
Before you commit, it's also worth understanding what a Shopify marketing agency actually costs — the pricing structures, what's included vs billed separately, and the math on when the fee is justified.
If you want to see what a Basic audit looks like before committing to anything, book a 20-minute call. No deck, no pitch.