AI, Bot and Crawler Policy
Updated: 9 October 2026
This policy forms part of the Terms of Service and Acceptable Use Policy. It explains how Noiz handles automated traffic on shared hosting.
Scope
This policy applies to Shared Hosting and Reseller Hosting. The numeric limits below describe the South African Plesk shared-hosting platform. Other hosting stacks require their own configuration; these values are not a claim about every Noiz server.
These defaults and limits do not automatically apply to Managed Virtual (VPS) or Dedicated Server environments. Where traffic threatens the stability or security of the network, protective measures may be applied while the situation is addressed.
What is allowed
Search engines, AI search and user-requested fetchers, AI training crawlers, SEO tools and other automated clients are permitted by default by Noiz's custom crawler policy, under the same shared-platform rate controls as ordinary visitors. This includes SE Ranking, Ahrefs and Semrush. There is no separate unlimited crawler tier. Three high-volume crawlers, Meta ExternalAgent, GPTBot and ClaudeBot, also share a separate platform-wide allowance, listed below. Identity is never used to raise a limit; for these three crawlers it lowers one.
A crawler is not blocked simply because its category is AI training or SEO, or because its address is absent from a published crawler list. User-agent names can be imitated and published address lists can be incomplete. Identity information helps investigate traffic. It does not grant an exemption from these rate controls, and a published crawler list or verified address does not change a rate.
Security protection remains active. Known abusive sources can be blocked or separately rate limited, and identified attack patterns are refused by the web application firewall. Direct add-to-cart or wishlist links opened without a referring page can be refused by those attack rules. Native firewall protections, a site's own security settings and any CDN or proxy in front of it can make separate decisions. Being permitted by this policy does not guarantee that every request will succeed. For example, Imunify360's own ByteDance crawler rule blocks Bytespider, and its own rate rules can refuse GPTBot and Meta ExternalAgent below the named-crawler allowance listed here. These vendor-managed protections remain enabled.
Why shared limits apply
Shared sites use common infrastructure. Rate controls constrain repeated requests, including legitimate crawling, and provide room for ordinary short bursts. Additional limits apply to selected shop-filter and cart-action requests because these have contributed to previous incidents.
A site can still reach its hosting plan's resource limits. These controls do not promise to prevent that, guarantee complete isolation from other sites, or guarantee search rankings or completion of an audit at its chosen speed.
Current shared-platform limits
| Request class | Sustained rate or concurrency limit | Burst allowance |
|---|---|---|
| General requests counted by the controls | 60 requests per minute per client key | 200 |
| Requests sent with the Meta ExternalAgent, GPTBot or ClaudeBot user agent | 30 requests per minute for each named crawler, one allowance shared across every site on the platform | 5 |
| Matched shop-filter and cart-action query parameters | 30 requests per minute per source address | 10 |
| Aggregate network traffic counted by the controls | 600 requests per minute per network key | 200 |
| POST requests to the standard WordPress login endpoint | 10 requests per minute per source address | 5 |
| Concurrent requests counted for one site | 200 across its clients | Not applicable |
Rate allowances are shared across sites on the same server, rather than renewed for each website. The general client key uses an IPv4 address. For IPv6, the current address matching groups addresses by /64 where supported and otherwise uses the individual address. The aggregate network key groups IPv4 by /24 and IPv6 by /48 where matched, otherwise using the individual address. People or crawlers sharing an address or network can share an allowance.
The named-crawler allowance is one counter per crawler for the whole platform, not one per site. A busy site can use a crawler's entire allowance, and Noiz does not schedule or guarantee crawl completion for any site. The allowance applies to GET and HEAD requests on every path except an exact request for /robots.txt. It is matched on the user-agent string only; it is a traffic control, not proof of who is sending the requests.
The shop-query limit applies to selected filter, sorting, cart, wishlist and comparison parameters. It is not a blanket limit on every cart, checkout, search or API page. Recognised WordPress administration and editor REST paths, and the WooCommerce REST API used by integrations, are excluded from that particular shop-query control.
Recognised static-file paths, robots.txt, sitemap paths and other platform control paths are excluded from the general, shop-query, network and concurrency counters. The named-crawler allowance excludes only an exact request for /robots.txt. Separate security controls and capacity constraints can still affect access.
Burst allowances accommodate short peaks; they do not increase the sustained rate or guarantee admission. Several controls may apply to one request. These rate and concurrency controls return HTTP 429 (Too Many Requests) when refusing a request. Legitimate visitors and crawlers can encounter a limit, particularly during bursts or behind a shared address.
Your site's choices
Your robots.txt belongs to you. You decide which compliant crawlers may access your content. Noiz does not use customer robots.txt files to enforce shared-platform limits. Documentation may recommend crawler settings, but robots.txt is advisory and does not replace access controls.
You can use your own application, CDN or site rules for stricter access choices. Shared-platform rate limits are mandatory and cannot be removed through a crawler allowlist or per-site opt-in. If you need different capacity or controls, contact support to discuss optimisation, an appropriate upgrade, a VPS or your own CDN rules. A shared-plan upgrade or CDN rule does not itself remove the origin's shared limits.
Keep your site maintained and reasonably efficient, as required by the Acceptable Use Policy.
Audits and false positives
SE Ranking and similar audit tools do not need, and cannot be given, a special allowance. Set the tool to no more than one request per second from one address, which is the general limit of 60 requests per minute. If an audit encounters persistent 429 responses, reduce its request rate or concurrency and retry later. A 403 can come from an attack rule, a site setting or an edge service; lowering the crawl rate is not a universal fix for it.
Report suspected false positives through support with the affected URL, approximate time and timezone, HTTP status, and source address or user agent if known. Noiz checks the relevant logs and corrects handling where warranted. Persistent search-crawler errors warrant investigation; a single refusal does not establish that a crawler is malicious.
Changes
This policy may be updated as the threat landscape and crawler ecosystem evolve. Material changes will be reflected on this page. In an emergency, protecting the integrity of the shared environment takes priority, as set out in the Acceptable Use Policy and Terms of Service.
