Better Robots.txt Pro

See how options shape the robots.txt file

Better Robots.txt Pro guides you through crawler settings, then generates a robots.txt file for review before you save it. Choose a preset, adjust individual modules using plain-language explanations, and inspect the resulting directives in the Review & Save step. Core WordPress protection rules remain visible and editable.

Set policies by crawler group

Manage search engine crawlers, AI training crawlers, AI search and discovery crawlers, SEO utilities, bad bots, archive services, feed crawlers, social preview bots, and other automated agents. Pro adds advanced controls such as per-bot AI rules, custom visitor agents, AI usage signals for search, ai-input, and ai-train, and optional machine-readable governance documents including llms.txt.

• Set crawler policies for archive and Wayback access, feeds, crawl traps, and social previews.
• Review WooCommerce crawl cleanup options and rules for CSS, JavaScript, and image crawling.
• Manage allowance for ads.txt and app-ads.txt, along with advanced directives such as crawl-delay and custom rules.

AI agents include training crawlers, search and discovery crawlers, and user-action fetchers. A robots.txt policy communicates preferences to crawlers that follow it; it does not force every bot to comply. User-action fetches may not follow robots.txt in the same way as automated crawling. Where available, stronger enforcement may require server logs, WAF rules, provider verification, or IP-based controls.

Select a preset, then tune its rules

Essential delivers a clean starting point for general websites. AI-First supports AI-related governance while keeping discovery available, and Custom lets you configure modules manually. Fortress is intended for stronger protection against scraping, archive capture, and unnecessary crawl activity, and is included with Premium. Preset and module availability is marked by edition in the interface. You can revisit settings and regenerate the file after your initial setup.

Configure and review in WordPress

Install and activate Better Robots.txt, then open its settings in your WordPress dashboard. Choose a preset or configure modules manually, follow the wizard, and review the generated robots.txt. Use Review & Save when the directives match your intended policy. You can return to the settings later to adjust them and generate an updated file.