Fork me on GitHub

Can I Crawl (this URL)

Hosted robots.txt permissions verifier.



Verifies if the provided URL is allowed to be crawled by your User-Agent. Pass in the destination URL and the service will download, parse and check the robots.txt file for permissions. If you're allowed to continue, it will issue a 3XX redirect, otherwise a 4XX code is returned.


$ curl -v

< HTTP/1.0 302 Found
< Location:
< HTTP/1.0 403 Forbidden
< Content-Length: 23

$ curl -H'User-Agent: MyCustomAgent' -v

> User-Agent: MyCustomAgent
< HTTP/1.0 302 Found
< Location:

Note: disallows requests to /search.


MIT License - Copyright (c) 2011 Ilya Grigorik