The AI Coin Flip: Why AI Gives Every Customer a Different Answer (Digital Reset Episode 488)
Thinks Out Loud · with Rand Fishkin (cited) · March 18, 2026 · 22 min
Summary
AI models like ChatGPT, Claude, and Google AI consistently deliver varied brand recommendations, even with identical prompts. This episode reveals how this inherent inconsistency impacts businesses using AI chatbots for customer interactions, highlighting that most teams haven
Key takeaways
Recognize that current generative AI models will produce inconsistent results (less than 1% identical output for the same prompt), which means every customer interaction with an AI chatbot can differ.
Shift focus from merely tracking brand visibility in AI outputs to understanding the broader implications of AI inconsistency across all customer-facing AI tools.
Develop monitoring frameworks that go beyond simple measurement, incorporating keyword tracking and sentiment analysis of AI-generated responses to manage variability.
Educate internal teams (marketing, sales, customer service) on the probabilistic nature of AI interactions to better manage expectations and strategize AI deployment.
Design AI strategies that account for multiple
good" answers rather than a single deterministic output, ensuring a range of relevant brand mentions even with varied AI responses.
Themes
ai & automationbrand & contentcustomer retentionanalytics & attribution
Rand Fishkin's team ran 2,961 prompts across ChatGPT, Claude, and Google AI. 600 volunteers, 12 different prompts, two months of runs. They wanted to answer one question: how often do you see the same list of brand recommendations twice, even with the exact same prompt?
The answer? Less than 1% of the time. The odds of seeing the same list in the same order are closer to one in a thousand.
Most conversations about AI inconsistency treat it as a measurement problem: how do I know if my brand is showing up? That's a legitimate question. But it's not the only question. And it might not even be the most important one. If AI systems give different recommendations essentially every time, the same inconsistency is already baked into every AI chatbot you've deployed — your hotel chat widget, your B2B sales assistant, your customer service tool. Most teams have never measured it. And some of those inconsistent answers are already drivi
Recognize that current generative AI models will produce inconsistent results (less than 1% identical output for the same prompt), which means every customer interaction with an AI chatbot can differ.
What does this episode say about brand & content?
Shift focus from merely tracking brand visibility in AI outputs to understanding the broader implications of AI inconsistency across all customer-facing AI tools.
What does this episode say about customer retention?
Develop monitoring frameworks that go beyond simple measurement, incorporating keyword tracking and sentiment analysis of AI-generated responses to manage variability.
What does this episode say about analytics & attribution?
Educate internal teams (marketing, sales, customer service) on the probabilistic nature of AI interactions to better manage expectations and strategize AI deployment.
