Robots.txt Generator

Use the form below to generate the content of robots.txt file, when you place that file on your server you'll be able to control access to specific search engine bots, whether you want to allow them access or deny it and even control their crawl speeds.

Robots access (All robots)	Crawl delay
Allowed (Default) Denied	No delay (Default) 5 seconds 10 seconds 20 seconds 40 seconds 80 seconds

Search engine	Access	Bot name
Google	Default Allowed Denied	googlebot
Yahoo	Default Allowed Denied	yahoo-slurp
Bing (MSN Search)	Default Allowed Denied	msnbot
Baidu	Default Allowed Denied	baiduspider
Alexa (Wayback)	Default Allowed Denied	ia_archiver
DMOZ	Default Allowed Denied	robozilla
Google Image	Default Allowed Denied	googlebot-image
Yahoo Blogs	Default Allowed Denied	yahoo-blogs/v3.9
Yahoo MM	Default Allowed Denied	yahoo-mmcrawler
Bing Pic-Search	Default Allowed Denied	psbot

Deny access to a specific directory
You must add a trailing "/" because path is relative to root. Example: /cgi-bin/
Restricted Directory:
Restricted Directory:
Restricted Directory:
Restricted Directory: