Robots.txt Generator and Tester
WEBTOOLCHEST.COM
ROBOT IT
?
GEN
TEST
ZIP
Generator
User Agents
Common bots…
Googlebot
Bingbot
GPTBot
CCBot
anthropic-ai
ClaudeBot
Slurp (Yahoo)
DuckDuckBot
facebookexternalhit
Twitterbot
Custom…
+ Add
Rules for *
+ Add Rule
Sitemap URL (optional)
Crawl Delay (seconds, optional)
Applies to the currently selected agent
Smart Mode — Zip
🔒
Smart Mode required
Drop your site zip to find, edit, and package your robots.txt directly.
Unlock Smart Mode →
Drop site zip here
Finds existing robots.txt and loads it for editing
✓ robots.txt found in zip — loaded above
Output — robots.txt
Click GENERATE to build your robots.txt
Tester
User Agent to Test
* (all bots)
URL Path to Test
Enter the path only — e.g. /admin or /sitemap.xml
— Awaiting input
Generate your robots.txt first, then enter a URL path above.
Or paste any robots.txt
Quick Reference
User-agent: *
— all crawlers
Disallow: /path
— block this path
Allow: /path
— allow this path
Sitemap: url
— sitemap location
Crawl-delay: n
— seconds between requests
Ready — add agents and rules, then click GENERATE
CLEAR
GENERATE
DOWNLOAD