Bots and AI crawlers now make up half of web traffic. How we handle them
About half of all web traffic is now automated, and by some measurements slightly more. Search engines, AI crawlers, monitoring tools and malicious bots together send as many requests as people do. AI crawlers in particular read large numbers of pages that nobody else has asked for, which makes each of their requests more expensive to serve. This article explains what the bots are, what robots.txt can and cannot do, how Noiz shares the load fairly on its hosting, and when a server of your own is the better fit.
Half of web traffic is no longer people
The security company Thales, which publishes an annual Bad Bot Report, measured automated traffic at 51% of all web traffic in 2024, the first time in a decade it passed human traffic. Its 2026 report puts the 2025 figure at 53%. Cloudflare, which carries a large share of the world's websites, found people and ordinary bots roughly level on web page requests by the end of 2025.
Not all of that is harmful. Search engines have to read your site to list it. But Thales counts about four in ten of all requests as coming from malicious bots: credential stuffing, scraping and probing for weaknesses.
Why a bot can cost more than a visitor
People tend to read the same popular pages. A website or its host keeps those pages ready in a cache, so serving them again is cheap. Crawlers behave differently. They work through a site systematically, including old posts, tag pages, search results and product filters that nobody has opened in months. Each of those pages has to be built from scratch: the site's code runs and the database is queried.
The Wikimedia Foundation, which runs Wikipedia, described exactly this in April 2025. Bots were about 35% of its page views but at least 65% of its most expensive traffic. Smaller organisations have felt it too. The documentation host Read the Docs reported one crawler downloading 73 TB in a single month in 2024, at a bandwidth cost of more than USD 5,000.
Training, search and fetch: three kinds of AI crawler
The large AI companies now publish separate crawlers for separate jobs:
| Kind | What it does | Examples |
|---|---|---|
| Training | Collects pages to train AI models | GPTBot (OpenAI), ClaudeBot (Anthropic) |
| Search | Builds an index so the AI's search can cite and link your page | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| User fetch | Opens a page because a person asked the assistant to read it | ChatGPT-User, Claude-User, Perplexity-User |
Training crawlers do most of the reading. Cloudflare found training crawling running at seven to eight times search crawling during 2025. They also send few visitors back. Cloudflare measured some AI crawlers reading tens of thousands of pages for every visitor they referred, against a handful of pages per visitor for Google's search crawler.
The search crawlers are different. Blocking them can keep your business out of the answers people now get from AI search tools, which is why Noiz allows them.
What robots.txt can and cannot do
robots.txt is a file at the root of your site that tells crawlers which parts they may read. Most of the large AI companies say their training and search crawlers follow it, and OpenAI, Anthropic and Perplexity say changes take effect within about a day.
It has limits:
- It is a request, not a lock. The robots.txt standard says plainly that its rules are not a form of access control.
- User-requested fetchers may not follow it. OpenAI says its rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page.
- Slowing down is patchy. A Crawl-delay line asks a crawler to pause between requests. Anthropic and Bing honour it; Google, OpenAI and several others do not document support.
- Not every bot is honest. Cloudflare reported in 2025 that one AI company's traffic switched to an ordinary browser identity when blocked. The company disputed the finding, but the point stands: a name in a request header is a claim, not proof.
How we share the load on our hosting
On shared hosting, every site on a server shares its capacity. Our approach, set out in the AI, Bot and Crawler Policy, is to keep the platform open and fair:
- Crawlers are allowed by default. Search engines, AI search, AI training crawlers and SEO tools can all read your site. We do not block a crawler because it is an AI crawler.
- Everyone works within the same rate limits. Crawlers and visitors share the same controls on how many requests one source can make in a minute. Short bursts are allowed for.
- The busiest AI crawlers get less, not more. Three high-volume crawlers share a separate, lower allowance across the whole platform. A crawler's name never raises a limit.
- Your robots.txt and sitemap are left alone. They sit outside the general limits, so search engines can always find their instructions.
- Attacks are refused. A web application firewall blocks known attack patterns, and known abusive sources can be blocked outright.
When a limit is reached, the server answers with "429 Too Many Requests", which tells well-behaved crawlers to slow down and try later. Google's own guidance treats a short spell of these as normal.
Your choices as a site owner
- Decide who may crawl. Your robots.txt belongs to you. You can allow search crawlers and disallow training crawlers, or the reverse.
- Keep the site efficient. Caching, a lean set of plugins and limits on endless filter combinations cut the cost of every request, from people and bots alike.
- Add Cloudflare. The Cloudflare add-on, added to a Noiz hosting plan from the client area, keeps your DNS synchronised to Cloudflare from the control panel, with managed Cloudflare at a predefined level: activation, deactivation, and basic configuration and security changes.
- Want to be found by AI search? Our help article on optimising WordPress for AI search covers llms.txt and what AI search tools look for.
When you want crawlers to read as much as they like
Some sites want every crawler to read everything as fast as it can: a large catalogue, a documentation site, a publisher that wants maximum reach in AI answers. Shared limits exist to protect neighbours, and they cannot be lifted for one site.
A Noiz Cloud server is the step up. Cloud 1 to 6 are self-managed virtual servers with full root access over SSH, Ubuntu or Debian, and a dedicated IPv4 address. The shared-hosting crawler limits do not apply: you set your own firewall and rate limits, and crawlers can read as much as your server can serve. You run the server yourself, so a new server owner should start with our guide to setting up a firewall on Ubuntu and Debian. If you would rather not run the server yourself, get in touch.
Sources
Checked 19 October 2026.
Traffic measurements
- Thales (Imperva), 2025 Bad Bot Report
- Thales, 2026 Bad Bot Report
- Cloudflare, 2025 Radar Year in Review
- Wikimedia Foundation, How crawlers impact the operations of the Wikimedia projects
- Read the Docs, AI crawlers need to be more respectful
- Cloudflare, Perplexity is using stealth, undeclared crawlers
Standards and crawler documentation
- RFC 9309, Robots Exclusion Protocol
- OpenAI, Overview of OpenAI crawlers
- Anthropic, Does Anthropic crawl data from the web?
- Perplexity, Perplexity crawlers
- Google, Google's common crawlers
- Google, Reduce the Google crawl rate
Noiz
Note: while some of this content was produced with the help of AI, the final editing has been done by a real human being.
