How AI reads your website: llms.txt, schema and entity clarity
AI systems read websites differently from humans: they parse served HTML, structured data and consistent naming, and they largely ignore visual design, animations and imagery. A website can win design awards and still be close to unreadable for the machines deciding whether to recommend the company behind it. This article explains what models and AI search engines actually see, layer by layer.
Layer 1: what your server sends to AI crawlers
The first question is brutally simple: when a crawler requests your page, what text comes back? Client-side JavaScript apps often return a nearly empty shell; the content humans see is assembled in the browser. Some AI crawlers execute little or no JavaScript, so everything they index is what your server sends. Server-side rendering of key pages is the single highest-leverage technical fix in most audits. The second question: are AI crawlers allowed at all? Blocking GPTBot, ClaudeBot or PerplexityBot in robots.txt removes you from those engines' answers - sometimes a deliberate choice, usually an accident.
Layer 2: structure and schema
Served text still needs interpreting. Clean heading hierarchy tells machines what each page is about; Organization, Product and FAQPage schema state it outright - name, category, offerings, people, links to profiles. Schema does not make weak content strong, but it removes ambiguity about entities, and ambiguity is what makes models hedge or skip you. llms.txt complements this: a plain-text index at your domain root pointing AI systems to your most important pages, cheap to publish and increasingly read.
Layer 3: entity consistency
Models cross-reference. Your website says one thing; LinkedIn, directories, review sites and press say others. When the category sentence, naming and key facts match everywhere, the model's confidence in recommending you rises; when they conflict - old positioning on a directory, a renamed product, two different company descriptions - the safest move for the model is to talk about someone else. Entity consistency is unglamorous and decisive.
EntityRise audits all three layers, fixes what fails and proves the change →
The failures that make companies invisible
The same handful of failures repeats across B2B audits: JavaScript-only content, blocked AI crawlers, missing or broken schema, claims locked in images and PDFs, and category language too vague to classify. None of them shows up in a design review, all of them show up in a measurement - which is why the fix always starts with running real buyer questions and recording what comes back.
Frequently asked questions
Does llms.txt actually matter?
It is a low-cost, low-risk signal: a growing set of AI systems reads it, publishing it takes an hour, and it never hurts. Treat it as cheap insurance, not a silver bullet.
Will schema markup alone get me recommended?
No single layer will. Schema removes ambiguity; recommendations still depend on evidence, buyer-intent content and citations. The layers compound.
How do I check what AI sees on my site?
Fetch your pages the way a crawler does (curl, or a rendering test) and read the raw HTML - or run a measured audit that tests machine readability alongside real buyer questions across ChatGPT, Claude, Gemini, Grok and Perplexity.
If machines cannot read your website, everything else you invest in AI visibility leaks.
EntityRise finds the failures and fixes them - measured before and after.