How AI search is changing SEO, and whether your site needs llms.txt
AI search engines answer questions by reading web pages and citing them, so being found now also means being quoted. The good news is that the work is the same as good search optimisation: clear, original pages that answer a question up front, can be indexed, and are open to the crawlers that feed search answers. Google says it needs no special files for its AI features and does not use llms.txt, which "will neither harm nor help". An llms.txt file is still cheap to add and useful to tools that read it. The decision that really matters is which crawlers you let in: those that power AI search, those that collect AI training data, or both.
What has changed
Google's AI Overviews and AI Mode, ChatGPT search, Claude and Perplexity answer a question in their own words and link to the pages they drew on. Google says its AI features may run several related searches behind the scenes and show a wider set of links than a classic results page.
For a business, that means two things. Pages that answer a question clearly at the start are more likely to be quoted. And a page an AI crawler cannot reach cannot be quoted at all.
What still works, and what Google says to skip
Google's guidance for its AI features is short:
- There are no extra requirements. A page needs to be indexed and eligible to show with a snippet.
- You do not need new machine-readable files, AI text files or special markup.
- You do not need to break content into tiny pieces for AI. Optimising for AI search "is still SEO".
In practice that means:
- Answer first. Put the answer in the first paragraph and the first sentence of each section.
- Write original material. Your own prices, procedures, experience and examples, not a rewrite of what is already out there.
- Keep sections self-contained, with headings that say what they contain.
- Date what changes, and change the date only when the page changes.
Our help centre has a practical guide to optimising WordPress for AI search.
What llms.txt is
llms.txt is a proposal, first published in September 2024, for a Markdown file at the root of a website that gives AI tools a short map of the site: its name, a one-line summary, and lists of links to its most useful pages. The proposal also suggests offering a clean Markdown copy of each page beside the normal one.
It is not a formal standard, and none of the main AI companies says its crawler reads it. Google says plainly that Search does not use it. It is still worth having when it costs little: some AI tools and agents fetch it when a person points them at a site, and a clean Markdown copy of each page is easy for any tool to read.
The Noiz help centre is an example. It publishes an llms.txt file listing every article, and every page is also available as Markdown.
Search crawlers and training crawlers are different
The AI companies now run separate crawlers for different jobs, and robots.txt lets you choose between them.
| Company | Feeds AI search or answers | Collects AI training data | Fetches a page when a user asks |
|---|---|---|---|
| Googlebot (Search, including AI Overviews) | Google-Extended (a robots.txt token, not a separate crawler) | ||
| OpenAI | OAI-SearchBot | GPTBot | ChatGPT-User |
| Anthropic | Claude-SearchBot | ClaudeBot | Claude-User |
| Perplexity | PerplexityBot (not used for training) | Perplexity-User | |
| Apple | Applebot | Applebot-Extended (a token, not a crawler) | |
| Common Crawl | CCBot, which builds an open web archive |
By the companies' own statements:
- Blocking a training crawler (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) keeps your future content out of model training without removing you from their search answers. Google says Google-Extended does not affect inclusion in Google Search.
- Blocking a search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot) reduces how often your site appears in those answers.
- User-triggered fetchers act for a person who asked about your page. OpenAI and Perplexity say robots.txt may not apply to them.
For most businesses that want to be found, the sensible default is to allow the search crawlers. Whether to allow training crawlers is a choice about your content, and either answer is reasonable.
How to set your choice
Your choices go in robots.txt at the root of your site. For example, to stay in AI search while opting out of training by OpenAI and Anthropic:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Remember that robots.txt is a request, not a lock. The standard, RFC 9309, says its rules are not a form of access control, and a crawler chooses whether to obey. Anything private belongs behind a login, not in robots.txt.
How Noiz handles crawlers
On Noiz shared hosting, search engines, AI search, AI training crawlers and SEO tools are all allowed by default, under the same rate limits as other traffic, and your robots.txt is yours to set. Our AI, Bot and Crawler Policy sets out the limits and how to report a crawler that is refused by mistake.
Questions about your site
If you have questions about AI crawlers or about how your Noiz hosting handles them, get in touch.
Sources
Checked 19 October 2026. Crawler names and policies change; these are as at October 2026.
llms.txt and search guidance
- llms.txt proposal
- Google: AI features and your website
- Google: AI optimisation guide
- RFC 9309: Robots Exclusion Protocol
Crawlers
- Google: common crawlers, including Google-Extended
- OpenAI: crawlers
- Anthropic: how Anthropic crawls the web
- Perplexity: crawlers
- Apple: about Applebot
- Common Crawl: CCBot
Noiz
- AI, Bot and Crawler Policy
- Noiz help centre llms.txt
- How to Optimise WordPress for AI Search (llms.txt and GEO) (Noiz help centre)
Note: while some of this content was produced with the help of AI, the final editing has been done by a real human being.
