Quick Presets

Visual Editor

User-Agent Rules

Sitemaps

Additional Options

Generated robots.txt

✓ Valid robots.txt

Common User Agents Reference

Search Engines

  • Googlebot - Google
  • Bingbot - Microsoft Bing
  • Slurp - Yahoo
  • DuckDuckBot - DuckDuckGo
  • Baiduspider - Baidu
  • YandexBot - Yandex

AI Crawlers

  • GPTBot - OpenAI
  • ChatGPT-User - ChatGPT
  • CCBot - Common Crawl
  • anthropic-ai - Anthropic
  • Claude-Web - Claude
  • Google-Extended - Google AI

Social Media

  • facebookexternalhit - Facebook
  • Twitterbot - Twitter/X
  • LinkedInBot - LinkedIn
  • Pinterest - Pinterest

Special Values

  • * - All bots
  • Googlebot-Image - Google Images
  • Googlebot-News - Google News
  • AdsBot-Google - Google Ads