Your robots.txt is a tool contract, and most sites ship it with no error handling. When an AI agent requests your pricing page and gets a 403 from your WAF, it does not file a ticket. It retries once, falls back to a cached snippet or a competitor's page, and moves on. You never see the failed call.
That is the frame for this post: treat every AI crawler hit as a function call with explicit failure states, and audit the ones you are silently failing. Three connections most GEO write-ups skip make the case.
1. Function calling failure states are your crawl logs
Reliable agents are built around typed results: success, retryable error, fatal error. Your site returns the same categories whether you designed them or not. A 200 with a JS shell and no server-rendered text is a success for the HTTP client and a fatal error for the model. A 429 from a bot-protection rule is retryable for the agent and invisible to your analytics.
Audit it like an API. Pull 30 days of logs, filter user agents (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot), and bucket responses by status and body size. Anything under about 2 KB on a content URL is probably a block page or an empty shell.
2. llms.txt is MCP-lite, and just as unenforced
The Model Context Protocol exists so an agent can discover capabilities from a manifest instead of guessing. llms.txt is the web's crude version: a markdown index telling a model which pages matter. The catch is that no major engine has confirmed it as a ranking input. Treat it as a manifest for agents that choose to read it, not a lever.
What it is good for is forcing you to decide your canonical set: 20 to 50 URLs with one-line descriptions. If you cannot write that list, the model cannot infer it from your nav either. Keep it in sync with your sitemap, and never list a URL that blocks bots.
3. Context windows reward the first 500 tokens
Large windows did not fix recall in the middle and at the edges of long inputs. Retrieval pipelines chunk your page and often embed the top chunks. So extractability is a layout problem: put the direct answer, the entity name, the price and the date in the first screen of rendered HTML. Burying them under a hero carousel, a cookie wall and 900 words of positioning means the chunk that gets retrieved is boilerplate.
Test it: fetch your page with curl and a bot user agent, strip scripts, and read the first 300 words. That is roughly what a retriever sees.
4. Closing the loop: measure what came back
Fixing access is half the job. The other half is checking whether it changed anything. That means tracking, per prompt and per engine, whether your brand is mentioned, which URLs are cited, and how that moves after each change. Doing this by hand across ChatGPT, Gemini, Perplexity, Claude, Google AI Overviews and AI Mode does not scale past a handful of prompts.
LLM Search Console does this: scheduled scans (manual, daily or weekly), share of voice and sentiment trends, competitor comparison by prompt, and a log of the sources each model cites with URLs and snippets. Ship a crawler fix on Monday, and you can see the citation delta by Friday instead of guessing.
Quick wins for GEO
Grep your logs for AI user agents and list every non-200 on content URLs.
Allow OAI-SearchBot, PerplexityBot and ChatGPT-User explicitly in robots.txt if you want citations. Blocking training bots is a separate decision.
Whitelist verified bot IP ranges in your WAF instead of relying on user agent strings.
Server-render the answer, price and entity name. Check with curl, not your browser.
Publish a short llms.txt of your canonical URLs and keep it matched to your sitemap.
Re-run your tracked prompts after every change and record the before and after.
Agree the failure is silent, and often it isn't in robots.txt at all. When I requested 30 big homepages as Chrome and as ChatGPT-User, 8 of the 23 that served Chrome normally refused the agent, and all 8 allow it in robots.txt. A log audit by user agent is the only way to catch that.