Cloudflare crawlers
Cloudflare may crawl or make HTTP requests to your site to make sure its protected and performing properly.
�� Crawling situations
Specific products
Cloudflare will crawl your site when you have specific products enabled:
- Always Online
- User-Agent:
Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)
- User-Agent:
- Health checks
- User-Agent:
Mozilla/5.0 (compatible; Cloudflare-Healthchecks/1.0; +https://www.cloudflare.com/; healthcheck-id: <HEALTHCHECK_ID>)
HEALTHCHECK_ID
is a 16-character string associated with the health check ID.
- User-Agent:
- Load balancing monitors
- User-Agent:
Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; pool-id: <POOL_ID>)
POOL_ID
is a 16-character string associated with the load balancing pool ID being monitored.
- User-Agent:
- Prefetch URLs
- User-Agent:
Mozilla/5.0 (compatible; CloudFlare-Prefetch/0.1; +http://www.cloudflare.com/)
- User-Agent:
- SSL/TLS recommender
- User-Agent:
Cloudflare-SSLDetector
- This crawler ignores your
robots.txt
file unless there are rules explicitly targeting the user agent.
- User-Agent:
- Security Insights
- User-Agent:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 (compatible; +https://developers.cloudflare.com/security-center/)
- User-Agent:
Other situations
Cloudflare will also crawl your site in other, specific situations:
- Speed tests
- User-Agent:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36 PTST/190628.140653
- Triggered when: You launch a speed test from within the Cloudflare dashboard.
- User-Agent:
- Support diagnostics:
- User-Agent:
Cloudflare-diagnostics
- Triggered when: Cloudflare Support Engineers perform error checks and by continuous monitoring used to raise intelligent alerts in the Cloudflare dashboard.
- User-Agent:
- Custom Hostname validation:
- User-Agent:
Cloudflare Custom Hostname Verification
- Triggered when: You choose to validate a custom hostname with an HTTP ownership token.
- User-Agent: