You cannot be cited by a system that cannot read you. Two technical mistakes quietly remove businesses from AI answers. The first is blocking the crawlers that feed AI answers, whether deliberately or through a default rule left over from a staging site. Each AI company runs more than one crawler, and they do different jobs.
OpenAI documents that OAI-SearchBot is the crawler that surfaces sites in ChatGPT search, while GPTBot collects content that may be used to train its models, and robots.txt controls the two separately.
Perplexity documents PerplexityBot as the crawler that surfaces and links sites in its search results. Google’s AI Overviews and AI Mode draw on the ordinary Search index, so Googlebot access is what counts there; Google-Extended governs use of content for Gemini training and grounding in some other Google systems, not inclusion in Search. You can decide on training separately, but never block the search crawlers.
Google also states that no special AI text file or schema markup is required to appear in its AI features, so treat an llms.txt file as optional housekeeping, not a lever. The second mistake is hiding content behind JavaScript. If your key answers only appear after a script runs, a crawler that does not execute that script sees an empty page. Put your important content in server-rendered HTML so it is present the instant the page is fetched. Both problems usually happen by accident. A developer ships a template with a blanket crawler block left on from a staging site. A modern framework renders the whole page in the browser, so the answer a human sees never reaches the crawler. Neither is malicious, and both make you invisible to the exact systems you want quoting you. That absence carries a growing cost:
Gartner predicted in February 2024 a
25 percent drop in traditional search engine volume by 2026 as AI chatbots and virtual agents absorb queries, so a business a crawler cannot read loses the citation and the click at the same time. Blocking crawlers to protect content is a false economy: it does not stop scraping so much as keep you out of the answer. Openness plus clean, server-rendered HTML is the price of being quoted.