ShipCircle
Launch

Free Robots.txt Tester and Validator

Test, validate and audit any website's robots.txt file. Verify crawl access for Googlebot, Bingbot, GPTBot, ClaudeBot, Perplexity and detect XML sitemaps.

Understanding Robots.txt & Crawl Directives

The robots.txt file is a plain-text document hosted at the root of your domain (e.g., https://example.com/robots.txt). It communicates with search engine spiders and web crawlers using the Robots Exclusion Protocol (REP), instructing them on which areas of your site they are allowed or forbidden to crawl.

Key Directives

  • User-agent: Identifies which crawler the directive applies to (e.g., Googlebot or * for all).
  • Disallow: Specifies URL paths that the robot must not request or crawl.
  • Allow: Explicitly permits access to a specific sub-path inside an otherwise disallowed folder.
  • Sitemap: Points crawlers directly to your XML sitemap URL for rapid URL discovery.

AI & LLM Crawlers

  • GPTBot: OpenAI's crawler used for training AI models and ChatGPT data retrieval.
  • ClaudeBot: Anthropic's crawler for powering Claude and model training datasets.
  • PerplexityBot: Perplexity AI's real-time crawler for conversational search citations.
  • Applebot: Apple's crawler for Siri suggestions, Spotlight, and Apple Intelligence.

Common Robots.txt Pitfalls to Avoid

  1. Blocking CSS and JavaScript: Never disallow access to your theme assets, styles, or scripts. Googlebot renders modern pages like a browser; blocking scripts prevents Google from evaluating mobile friendliness and layout.
  2. Using robots.txt to hide sensitive data: Robots.txt is publicly visible to everyone. Disallowing a secret admin path simply advertises its location. Use server authentication (HTTP auth) or session logins instead.
  3. Expecting Disallow to de-index pages: Disallowing a page tells crawlers not to crawl it, but Google can still index the URL if other sites link to it. To remove a page from search results, allow it to be crawled and include a <meta name="robots" content="noindex"> tag.

Need to test your page snippets or meta tags? Use our Google SERP Simulator or check on-page headings and schema with the On-Page SEO Checker.

How the Robots.txt Tester works

  1. Enter a domain or URL. We fetch https://example.com/robots.txt directly from the host root.
  2. We check HTTP status, parse user-agent blocks, and evaluate access for major search engines and AI crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot).
  3. We detect XML sitemaps, highlight syntax warnings, and provide an interactive path tester to check if specific URL paths are allowed or blocked.

Frequently asked questions

Where should robots.txt be located?

The robots.txt file must always reside at the root of your domain (e.g., https://example.com/robots.txt). Files placed in subfolders are ignored by search engines.

What is the difference between Disallow and Noindex?

Robots.txt Disallow stops crawlers from downloading the page. However, the URL can still appear in search results if linked elsewhere. To keep a page out of search indexes completely, use a 'noindex' meta tag instead.

Should I block AI bots like GPTBot or ClaudeBot?

Blocking training bots prevents companies from using your content for model training. If you want your content cited in AI answer engines (like ChatGPT Search), make sure search crawlers like Googlebot and OAI-SearchBot are permitted.

Is the robots.txt tester free?

Yes. The tester is free to use with no account required.

More free tools