Case Study Methodology

What counts as a result

A marketing result is a change in a metric that was measured before the work started, using data the client already had. Anything short of that is an anecdote.

The distinction matters because the alternative is easy: describe a project in general terms, imply an improvement, and never state a number that could be checked. It reads well and it means nothing.

The second column of the right-hand side is not a high bar. It is the minimum for a number to mean anything, and it is the standard we report to.

What evidence we require before reporting

Four conditions, all of which must hold before a result appears in any material we publish. They are deliberately strict, because the alternative is a number that cannot survive scrutiny.

1. A baseline exists before work begins. Measured on the client's own analytics, in their own account, before any change. A baseline established afterwards is not a baseline.
2. The measurement window is stated. Start date, end date, and the comparison period. A percentage without a period is not a result.
3. The data source is named. Search Console, the ad platform, the analytics tool, the CRM. The reader must be able to check the same number you did.
4. The attribution model is named. Last-click, data-driven, or a stated manual model. This single choice can move a reported return by a factor of two, so it is never left implicit.
5. Other significant changes are disclosed. A pricing change, a product launch, a new distribution deal or a site rebuild during the period all affect the result. A result reported without them is misleading even if every number is correct.

What a case study contains

A case study that meets the standard above has six parts. Any account missing more than two of them is an advertisement, not a case study.

How to verify an agency's numbers

Useful regardless of who you are asking, and the fastest way to tell the difference between a consultancy and a sales operation.

• Ask what the metric was before the work started. A number without a baseline is a feeling.
• Ask which tool the data came from, and whether you can see it yourself. An agency that will give you account access has nothing to hide.
• Ask what the attribution model is. If they cannot name it, the return figure is not meaningful.
• Ask for the period. "We improved their results" over an unspecified window can mean anything.
• Ask what else changed during that period. The omission is almost always accidental and always material.
• Ask for a client whose engagement did not go well. An agency that only shows successes is showing you a portfolio, not evidence.
• Ask what they would do differently. The answer quality tells you more than any metric on the page.

The last two are the most revealing and the least often asked. An agency willing to discuss a failed engagement is telling you that it measures outcomes rather than managing appearances.

Results that did not work

Some engagements produce a result that is not the one the client wanted, or no result at all. Two things are true about that, and both matter.

First, a negative result is genuinely informative. A well-run test that shows a channel is not viable for your business saves considerably more than a year of funding it. The value of the work is in the finding, not the outcome.

Second, it is far more common than marketing material implies. Most engagements produce a mixed result, and a serious consultancy will say so when asked. A case study presenting an unbroken run of successes is telling you the accounts have been selected, which is legitimate — but it means you are not seeing the average outcome.

How to start an engagement

Every engagement begins the same way: with measurement. If the baseline does not exist, it is established first, because without it no future result can be reported to the standard above.

From there the sequence is consistent: diagnose what is actually limiting growth, agree the metric that represents it, fix the highest-leverage constraint, and report against the agreed standard throughout rather than at the end.