Short answer: Your agency is working if your own sales data shows more new customers or qualified pipeline at an acquisition cost your margins can carry, and that growth would not have happened without it. Judge it on revenue, new customers, CAC and contribution margin from Shopify or your CRM, not on the ROAS, clicks and impressions in the agency's report.
Key takeaways
- Outcome metrics prove an agency is working: total revenue, new customers, new customer CAC, MER and contribution margin. For B2B, qualified leads, pipeline and closed revenue.
- Platform-reported conversions are almost always higher than reality because of attribution windows, view-through conversions, double counting, branded search and retargeting people who would have bought anyway.
- Reconcile every month: compare what Meta and Google claim with what Shopify or your CRM actually recorded, split by new and returning customers.
- A simple pause test or geo split tells you whether the spend is causing sales, not just standing next to them.
- Give paid search and paid social about three months and SEO six to twelve before a final verdict, but some red flags tell you something is wrong much sooner.
- Run the same short monthly review from your own numbers, so your verdict rests on a trend rather than one good or bad month.
Once the contract is signed, the question changes: is the money going out each month bringing more money back in? The difficulty is that most agency reports are built on the numbers ad platforms produce, and those numbers are designed to make advertising look effective.
This guide is for owners who already have an agency in place and want a fair, evidence-based way to judge it. Whether a channel can pay for your business at all is a separate question, which we cover for search in are Google Ads worth it. Here the question is whether your agency is making it pay.
Which KPIs actually prove a marketing agency is working?
The KPIs that prove an agency is working are the ones that show up in your own systems and in your profit: total revenue, number of new customers, cost to acquire each new customer, and the margin left after marketing. Everything else is supporting evidence at best.
It helps to split marketing agency KPIs into layers and be clear which one you judge on.
Outcome metrics: what you judge the agency on
- Total revenue from your shop or finance system, compared with the same period last year and the months before the agency started.
- New customers: the count of first-time buyers. Growth that only comes from existing customers is usually email, loyalty or brand, not acquisition.
- New customer CAC: total paid marketing spend (plus agency fees if you want the honest version) divided by new customers. This is the single most useful number for most ecommerce brands.
- MER (marketing efficiency ratio): total revenue divided by total marketing spend across every channel. It ignores attribution entirely, which is exactly why it is useful as a cross-check.
- Contribution margin after marketing: revenue minus product costs, shipping, payment fees, ad spend and agency fees. If this is not rising, growth is not paying for itself.
For B2B: pipeline, not leads
For lead generation the equivalent outcomes are qualified leads (accepted by sales, not just form fills), sales opportunities created, pipeline value, closed revenue and cost per opportunity. Lead counts on their own are easy to inflate with cheap forms, broad targeting or spam. If your agency reports leads but has never seen your CRM, they cannot tell you which campaigns produced the deals.
Leading and activity metrics: useful, but not proof
Click-through rate, cost per click, impressions, sessions, rankings and platform ROAS are diagnostics. They help the agency find what to fix, but a rising CTR with flat new customers means the ads got more interesting, not that they made you money.
The common mistake is agreeing to a dashboard of twenty metrics without naming which two or three decide whether the relationship is working. Pick them together, write them down, and agree on where each one comes from. A performance marketing agency that is confident in its work will usually welcome being judged on outcomes.
Why do the platform numbers look better than your actual sales?
Because each ad platform counts any sale it can plausibly connect to its ads, using its own rules, and none of them checks with the others. Add up what Meta, Google and TikTok each claim and you will often get more revenue than your shop actually took.
Attribution windows
Platforms credit a sale to an ad if the buyer clicked or saw it within a set window. Someone who clicked an ad on Monday, forgot about it, then searched your brand on Friday and bought counts as an ad sale. Longer windows mean bigger numbers, and a window changed silently between reports makes the trend meaningless.
View-through conversions
Meta can count a purchase if the person merely saw an ad and later bought, without clicking. If your loyal customers scroll Instagram daily, many buyers will have seen an ad at some point without it causing anything. Ask what share of reported conversions are view-through, and judge the click-based figure separately.
Double counting across Meta and Google
A buyer sees a Meta ad, later clicks a Google Shopping ad, then buys. Meta claims the sale. Google claims the sale. Your shop recorded one order. Each platform's report is internally consistent and the combined total is wrong.
Branded search
People who type your brand name into Google were mostly going to find you anyway, so branded campaigns show very high ROAS on demand that already existed. There can be good reasons to run them, such as competitors bidding on your name, but report them separately and never use them to prop up the account average.
Retargeting existing customers
Retargeting and broad campaigns frequently show ads to people who have bought before or were already in the checkout. When they buy again, the platform claims it. If your agency's best campaign is mostly reaching your own email list, it is taking credit for revenue your email and brand would have earned.
Shift budget towards branded search and retargeting, lengthen attribution windows, and platform ROAS will usually go up. None of those changes brings in a single extra customer. If ROAS improves but new customers and total revenue are flat, ask what changed in the account.
None of this means the agency is being dishonest. Platform reporting answers "which ads were near a sale?", while you need to know "which sales would not have happened without the ads?"
How do you check the agency's report against your own data?
Each month, put the platform-reported numbers next to what Shopify, your ecommerce platform or your CRM actually recorded, split by new and returning customers, and calculate CAC and MER from your own totals. It takes about an hour once the template exists, and it only works if you have your own login to the ad accounts, analytics and sales data rather than screenshots from the agency.
Here is a worked example for a fictional online shop. The numbers are illustrative, but the pattern is one we see often.
-
€74,000Revenue claimed by platforms
Meta €42,000 plus Google €32,000 on €18,000 spend -
€80,000Total revenue in Shopify
All channels, including email and organic -
350New customers
€28,000 of revenue from first orders -
€51New customer CAC
€18,000 spend divided by 350
The agency report says ROAS is 4.1. That looks excellent. But the platforms are claiming 92 percent of all revenue, while the shop also has email, organic search, direct traffic and repeat buyers who were buying before the agency arrived. Something does not add up.
What the platforms said
Meta ROAS 4.2, Google ROAS 4.0, combined €74,000 attributed. Meta includes €11,000 from one-day view-through. Google includes €14,000 from brand search campaigns.
What the shop said
€80,000 total revenue. €52,000 came from returning customers, many of whom also received email. Only €28,000 came from 350 first-time buyers.
What it means
MER is 4.4 (€80,000 / €18,000), which looks healthy. But with an €80 average first order and 60 percent gross margin, each new customer brings €48 of margin on the first order against a €51 CAC. Acquisition roughly breaks even and only pays off if customers come back.
Breaking even on the first order can be a sensible strategy if repeat rates are good. But it is a very different story from "ROAS 4.1", and it leads to better questions: is the brand search spend needed, is Meta finding new people or recycling existing customers, and what is the 90-day repeat rate of this month's new customers?
How to set up the monthly reconciliation
- Export total orders, revenue and new versus returning customers from Shopify (or deals and revenue by source from your CRM) for the calendar month.
- Pull total spend from every ad account, and the agency's reported conversions, split into branded and non-branded, click and view-through.
- Calculate MER, new customer CAC and contribution margin after marketing from your own figures.
- Compare with the previous three months and the same month last year. Look at the trend, not the single month.
- Note the gap between platform claims and actual totals. A stable gap is fine. A widening gap means more of the reported growth is attribution, not sales.
For B2B, match platform-reported leads against your CRM and follow them to opportunities and closed deals.
Marketing analytics services that reconcile ads with real sales
We set up tracking, a source-of-truth dashboard and a monthly reconciliation built on your Shopify or CRM data, so you can judge any agency, including us, on what actually happened.
See our marketing analytics servicesHow can you test whether the agency is actually causing the sales?
Switch the activity off somewhere, for a while, and see how much revenue disappears. That is the core of an incrementality test. It can be as simple as pausing a campaign type for two to four weeks, or turning ads off in some regions while keeping them on in others.
The pause test
Pause one clearly defined thing, such as branded search or retargeting, for two to four weeks, and watch total revenue and new customers in your own data. If revenue barely moves, that campaign was mostly taking credit. If it drops by roughly what the campaign claimed, it was genuinely adding sales.
Pause tests are cheap but noisy. Avoid sales, launches and seasonal peaks.
The geo split
A geo split is more reliable. You pick regions that normally behave similarly, keep ads running in some and switch them off in others, then compare.
Before the test
Over the previous eight weeks, region B consistently sold €25 for every €100 sold in region A. That ratio is your baseline.
During the test
Meta ads stay on in region A and are switched off in region B for four weeks. Region B now sells €20 for every €100 in region A.
What it tells you
Without ads, region B lost €5 of every €25, about 20 percent of its sales. If Meta was claiming 50 percent of revenue, the true contribution is much lower than reported, and budget decisions should use the tested figure.
Some platforms offer their own lift studies using randomly held-out audiences. They are worth using, but confirm the result in your own sales data.
The common mistakes: testing with too little volume, so normal variation drowns out the effect; stopping before two weeks; changing prices or email schedules mid-test; and treating one result as permanent. Repeat key tests once or twice a year.
A good agency will propose and run these tests with you. Reluctance to test is itself information.
How long should you give an agency before judging the results?
For paid search and paid social, expect a meaningful early read after four to eight weeks and a fair verdict after about three months. SEO typically needs six to twelve months before revenue impact is clear. B2B needs at least one full sales cycle on top of that.
These are not periods in which an agency may show nothing. Every channel should show directional progress well before the final verdict.
| Channel | Early signal to expect | Fair point to judge outcomes |
|---|---|---|
| Google Ads and PPC | Cleaner account structure, wasted spend cut, tracking fixed within 2 to 6 weeks | About 3 months |
| Meta and paid social | Creative tests running, winners identified within 4 to 8 weeks | About 3 months |
| Email and automation | Core flows live, revenue per recipient tracked within 4 to 8 weeks | 2 to 4 months |
| SEO and content | Technical fixes shipped, impressions and rankings moving at 3 months | 6 to 12 months |
| B2B lead generation | Qualified lead rate and sales feedback within 6 to 8 weeks | One full sales cycle plus about a month |
The mistake runs both ways: firing a paid social agency after six weeks because the first creative tests lost, or tolerating two years of vague SEO ranking reports because "SEO takes time". Agree what progress should look like at each checkpoint.
Red flags that do not need a full quarter
- Reports lead with impressions, reach and CTR, and never mention new customers, CAC or pipeline.
- Spend keeps rising while new customers in your own data stay flat.
- Branded search or retargeting makes up a growing share of the reported results.
- The agency cannot explain what it changed last month, why, and what happened as a result.
- Every report is a win. Real accounts have tests that fail. An agency that never shows one is either not testing or not telling you.
How do you turn these checks into a verdict?
Put them into one short monthly review built from your own numbers, and judge the agency on the trend across three or more months, not on any single report. Send it to the agency a day before your call, so the discussion starts from your data rather than theirs.
The agency's response to that review is evidence too. An agency that is working can explain the gap between its report and your totals, names the tests that lost, and tells you what it will change and how you will both know whether it worked. One that answers flat new customers with rising ROAS and "attribution is complicated" is asking you to judge it on its own numbers. If you want more questions to put to it, our guide to questions to ask a digital marketing agency includes several that are just as useful mid-contract as before signing.
Frequently asked questions
There is no universal good ROAS, because it depends on your margins, repeat rate and how the platform counts conversions. A better target is a new customer CAC your first-order or 90-day margin can support, checked against your own sales data.
Calculate it both ways. CAC on ad spend alone is useful for comparing campaigns and channels. CAC including agency fees, tools and creative costs is the honest number for deciding whether the whole investment pays off.
Each tool uses its own attribution model, lookback window and tracking method. Analytics usually gives credit to one session based on its own rules, while each ad platform credits itself for any sale it touched. Expect differences, and use your shop or CRM totals as the final reference.
Yes, though with less precision. A simple pause test on branded search or retargeting over three to four weeks, during a quiet period, is enough to spot campaigns that are mostly taking credit. Geo splits need more volume, so they suit businesses with steady daily orders across several regions.
No. Platform numbers will never match your shop exactly. What matters is that the agency acknowledges the gap, explains it, reports your own totals alongside its platform numbers, and keeps the gap broadly stable rather than letting it widen.
What to do next
You do not need a data team to judge your agency fairly, just your own numbers and a consistent monthly check.
- This week, pull last month's revenue, new customers and total ad spend from your own systems and put them next to the agency's report.
- Turn that into a one-page monthly review using the checklist above, starting with new customer CAC and MER.
- Agree with your agency on two or three outcome metrics, and propose one pause test or geo split for next month.
If you would like an independent view first, we offer a free written audit within 48 hours covering your tracking and how your reported results compare with your actual sales.