Structiq
← All tools

Robots.txt & Sitemap Tester

Check whether a site exposes robots.txt and sitemap files correctly and whether AI/search crawlers can access the site.

Tool execution is available for logged-in users and visitors who've completed a free StructIQ scan.

Run Free ScanLog In
Free account required(unlocked after your first free scan)

Why robots.txt & sitemaps matter for AI visibility

robots.txt controls which parts of your site crawlers can access, and sitemap.xml lists the URLs you care about most. If either is missing or misconfigured, search engines and AI crawlers may struggle to discover and trust your content.

This tester checks whether those files exist, how they're wired together, and how common AI/search bots (ChatGPT, Claude, Bing, Google) are treated. That context directly affects whether you're even in the pool of candidates for AI citations.

Where these files live

  • robots.txt – https://yourdomain.com/robots.txt
  • sitemap.xml – typically /sitemap.xml or a sitemap index.
  • The robots file can reference one or more sitemaps using a Sitemap: line.

For managed platforms, you may need to edit robots.txt via hosting settings or theme configuration instead of uploading a plain file.

How to use these results

  • If robots.txt is missing, work with your dev/host to add one that allows core crawlers to reach your content.
  • If the sitemap is missing or not referenced, add a sitemap and expose it inside robots.txt.
  • Review AI bot statuses — blocking search bots (e.g. ChatGPT Search, Claude Search) can sharply limit AI visibility.

Safe robots.txt updates

  • Be careful with Disallow: / or broad disallows—those can de‑index your site from both search and AI.
  • Allow read‑only AI/search bots to access public content while still blocking training bots if that's your policy.
  • Keep sensitive paths (admin, checkout, internal APIs) disallowed as needed.
  • Re‑run this tool after changes to confirm AI and search bots still have the right level of access.

Want to know if AI can cite your site?

Run a free AI visibility scan and see how your website performs in AI search systems like ChatGPT, Claude, and Gemini.

Run Free AI Visibility Scan →