The single biggest thing that’s changed in vetting agencies is how easy it’s become for any firm to look incredible on the surface. With modern design tools and AI assistance, a mediocre agency can produce a stunning capabilities deck and seemingly impressive case studies overnight. As Kevin Simonson explained on The eCommerceFuel Podcast, this makes it incredibly difficult for a brand owner to distinguish a genuinely effective partner from one that just talks a good game. The result is that you have to be more skeptical than ever and shift your evaluation process from what they present to what you can independently verify.
This means that the case study, once the gold standard for agency vetting, is now just the starting point. You can’t take it at face value. When an agency shows you a chart of massive returns, your first question should be about the inputs. What was the ad spend? What was the attribution window? Was this a one-off viral success or a sustained, repeatable strategy? As the hosts of Ecommerce Playbook often discuss, the numbers don't tell the whole story without the context. If an agency is cagey about these details or presents only vague percentages, it’s a major red flag. A great partner will be excited to open the hood and show you the engine, not just the shiny paint job.
This is why talking to references is no longer optional; it’s the most critical step in your due diligence. I’d go so far as to say if an agency can't or won't connect you with at least two current or recent clients, you should walk away. No exceptions. When you get on the phone with a reference, you’re not just confirming the case study. You’re asking about the relationship. How is their communication? What happens when a campaign doesn't work? How do they handle reporting? This is where you’ll uncover the truth about what it’s like to work with them, which is something you'll never find in a pitch deck.
Ultimately, the big shift is from buying a deliverable to hiring a team. Andrew Faris puts it best on Ecommerce Playbook when he says his number one rule is to “Hire Smart People.” You are not hiring the case study or the proposal document. You are hiring the brains, experience, and problem-solving skills of the people who will be working on your account. Your interview process should focus on them. Ask them to diagnose a problem your business is facing. Give them access to your ad account and see what they find. The goal is to get past the polish and see how they think. Their thinking is what you’re paying for.
Finally, this rigor should extend to the contract itself. On The eCommerceFuel Podcast, Kevin Simonson made a great point about aligning the agency’s fee structure with your own goals. A great agency wants to win when you win. If they are pushing for a long-term retainer with no performance incentives, it’s worth asking why. A structure that includes a base fee plus a performance component often creates a healthier, more aligned Client-Agency Relationship. It shows they have skin in the game and are confident in their ability to deliver real, measurable results for your business.