Fake crawlers: verify identity before changing nginx rules

A crawler name in a User-Agent header is a claim. Before blocking traffic, separate three questions: who sent it, which paths it requested and what response it received. A cloud network or a fast request burst does not answer all three.

Current provider ranges can help classify current requests; they do not establish the identity of every request in an older log window. Keep historical traffic unconfirmed when its identity evidence is incomplete.

Keep claims and verification in separate columns

A useful log review starts with the time window, timezone and site host. Shared-server totals can otherwise be mistaken for one site’s traffic. Keep a request’s claimed bot name beside its source address, path, status and identity-verification result.

Window / timezone / site host:
Claimed user agent:
Source address / trusted proxy handling:
Provider category / verification method:
Provider list or DNS evidence / retrieval date:
Identity: VERIFIED / UNCONFIRMED / CHECK FAILED
Requested path / response status:
Behaviour finding / evidence:

Do not publish source addresses, private paths or credentials from operational logs. Aggregate the result and retain the underlying evidence privately. A verification timeout leaves identity unconfirmed; it does not establish malicious intent.

Verify against the provider’s current method

For Google requests, use the documented crawler or fetcher category. Common crawlers, special-case crawlers and user-triggered fetchers have different published ranges and hostname patterns. Google documents user-triggered fetching from Google Cloud as well: cloud hosting alone is not a counterfeit test.

For a DNS check, reverse-resolve the source address, check the hostname against the provider’s documented domain pattern, then forward-resolve that hostname and confirm the original address is present. Forward and reverse agreement without the provider-domain check proves only that those DNS records agree. Match domain boundaries carefully rather than accepting a name that merely contains a provider’s name.

For larger reviews, match the address against the appropriate official published range list and save the list retrieval date. An ASN identifies a network, not a particular crawler service. Do not reuse Google’s verification rules for another provider; consult that provider’s own documentation.

The User-Agent remains useful for grouping claimed traffic. It is not authentication, and an address outside a current list is not by itself proof that a historical request was spoofed.

Separate suspicious paths from identity findings

Requests for environment files, repository metadata or debug endpoints deserve investigation regardless of the claimed crawler name. Check the response body safely as well as the status. A200 might be the ordinary homepage returned for an unknown query; a 404 might simply be an absent page. Neither code alone proves a leak or a genuine crawler visit.

Report 2xx, redirects, 304 and errors separately. Also distinguish a blocked request from a crawler that never visited. No confirmed visits means the access rate is not measured, not that access is blocked.

Tune nginx from observed workload

Rate limiting can protect a service, but there is no universal human-versus-bot threshold. Shared addresses, asset bursts, crawlers and legitimate integrations can all exceed an arbitrary per-address limit. Establish the protected resource, inspect observed demand and test the effect on ordinary users before applying a policy.

Keep page requests and static-asset bursts distinguishable. Check the effective inherited nginx configuration rather than assuming one location block disables a parent limit. Review IPv4 and IPv6, trusted proxies and the rollback path with the operator. This article supplies a review method, not a production access-control configuration.

Do not block broad cloud ranges on the assumption that official fetchers never use them. Test provider verification and behaviour rules separately, then confirm that legitimate search and answer-engine requests still reach the intended public pages.

Sources

Checked15 October 2026: Google’s crawler and fetcher verification methods. The linked category lists are the reference for Google identity checks; this article does not maintain a duplicate range list.