Insights · GEO

The three files an AI engine reads before it reads your homepage.

Romaan Sheikh, Director · September 2026 · 6 min read

When ChatGPT, Claude or Gemini decides whether your business exists, it does not start on your homepage. It starts with three plain-text files at the root of your domain: robots.txt, sitemap.xml and, increasingly, llms.txt. Most businesses have never opened them. All three are visible to anyone with a browser, and all three of ours are linked at the end so you can read what a deliberate set looks like.

robots.txt: the door

This file tells crawlers what they may read. It is where AI crawlers are most often locked out by accident: a hosting platform or CDN switches on a "block AI bots" default and the file is rewritten without anyone deciding. Open yourdomain.com/robots.txt and look for the names GPTBot, ClaudeBot, Google-Extended and PerplexityBot. A line reading Disallow: / under any of them means that engine cannot read you, and will describe you from what other sites say instead. That is a decision worth making on purpose. For most small businesses, being described by third parties costs more than being read directly.

The file should also point at the sitemap, one line, so a crawler that arrives at the door knows where the rooms are.

sitemap.xml: the map

A sitemap is a list of every page you want indexed, with the date each one last changed. Engines use it to decide what to crawl and how often to come back. Three faults are common. Pages that no longer exist are still listed, so the crawler wastes its visit on errors. Pages you would rather nobody read, a staging copy, an unlisted offer, are listed because the platform generated the file automatically. And the last-modified dates are all identical, usually the date the site was rebuilt, which tells the engine nothing about what actually changed. A sitemap that is hand-checked, or generated from the same source that writes the pages, avoids all three.

Test yours by opening yourdomain.com/sitemap.xml and clicking every entry. If any of them return an error page, the file is lying to the crawler.

llms.txt: the summary

This is the newest of the three and the least settled. The proposal is a plain-text file at /llms.txt that states, in a few hundred words, what the business is, who it serves, what it charges and where the key pages are, written for a language model rather than a person. Honest position: the major engines have not publicly committed to reading it, and we do not claim it moves any measurable needle today. We publish one because it costs half an hour, because it forces the business to state its own facts in one place without adjectives, and because if adoption arrives the file is already there. Write it as if a careful stranger will quote it verbatim, because one day one might.

Why files before pages

A homepage is written for a human who can be persuaded. The three files are written for a machine that can only be informed. An engine that reads "award-winning solutions for ambitious brands" learns nothing. An engine that reads a company name, a registered address, a list of services with prices and a set of live pages learns exactly what to say when someone asks who does marketing for a fostering agency in the north of England. The files are where that plain statement lives, and the homepage is downstream of it.

Ours, for the record

You can read all three at robots.txt, sitemap.xml and llms.txt. They are maintained alongside every page on this site and re-checked whenever a page is added, which is how this article came to be listed in the map before it was published. The full checklist we run, including the trap we caught on our own domain, is in the GEO checklist.