How AI reads your website: llms.txt, schema and entity clarity

What does your server send to AI crawlers?

The first question is brutally simple: when a crawler requests your page, what text comes back? Client-side JavaScript apps often return a nearly empty shell; the content humans see is assembled in the browser. Some AI crawlers execute little or no JavaScript, so everything they index is what your server sends. Server-side rendering of key pages is the single highest-leverage technical fix in most audits. The second question: are AI crawlers allowed at all? Blocking GPTBot, ClaudeBot or PerplexityBot in robots.txt removes you from those engines' answers - sometimes a deliberate choice, usually an accident.

How do structure and schema help AI interpret your site?

Served text still needs interpreting. Clean heading hierarchy tells machines what each page is about; Organization, Product and FAQPage schema state it outright - name, category, offerings, people, links to profiles. Schema does not make weak content strong, but it removes ambiguity about entities, and ambiguity is what makes models hedge or skip you. No major AI provider has publicly confirmed actively reading llms.txt; Ahrefs' June 15, 2026 study found adoption is driven by speculation rather than provider confirmation (Ahrefs study, published June 15, 2026). Google Search has stated it ignores llms.txt; the same study summarizes Google's Search guidance as saying site owners can skip the file (Ahrefs study, published June 15, 2026).

How often do llms.txt files actually get traffic?

Ahrefs' June 15, 2026 analysis of 137,000 domains found that 97% of valid llms.txt files in its sample received zero requests in May 2026 (Ahrefs study, published June 15, 2026). Ahrefs cautions in that June 15, 2026 study that a fetched file is not necessarily read or used, so the share that influences an AI answer may be smaller still.

Where can llms.txt still be useful?

The file is most relevant when a developer tool, code assistant or custom agent is explicitly built to consult it. Ahrefs' June 15, 2026 traffic analysis found more requests from agentic and coding tools than from AI retrieval bots (Ahrefs study, published June 15, 2026). Treat llms.txt as optional documentation for those tools, not as a general AI-search ranking signal.

Do your company details match across the web?

Models cross-reference. Your website says one thing; LinkedIn, directories, review sites and press say others. When the category sentence, naming and key facts match everywhere, the model's confidence in recommending you rises; when they conflict - old positioning on a directory, a renamed product, two different company descriptions - the safest move for the model is to talk about someone else. Entity consistency is unglamorous and decisive.

The $49 audit shows what the five assistants actually say about your company, so your team knows which layer to fix first →

Which failures make companies invisible to AI?

The same handful of failures repeats across B2B audits: JavaScript-only content, blocked AI crawlers, missing or broken schema, claims locked in images and PDFs, and category language too vague to classify. None of them shows up in a design review, all of them show up in a measurement - which is why the fix always starts with running real buyer questions and recording what comes back.