AI crawler and llms.txt checker
Enter your website. See which AI crawlers your robots.txt lets in, whether your llms.txt is there and readable, and whether your firewall turns AI crawlers away.
What this means and how to fix it
ChatGPT, Claude, Perplexity, and Gemini can only cite pages their crawlers are allowed to read. The ones that matter most for being named in answers are the search crawlers: OAI-SearchBot, PerplexityBot, and Bingbot. If any of them shows Blocked, find the line shown in your robots.txt and remove it, or ask whoever manages your website to.
Training crawlers such as GPTBot, ClaudeBot, and CCBot are a business choice. Blocking them keeps your pages out of model training; it does not remove you from search answers.
If the firewall check flags a problem, look at your host's or Cloudflare's bot settings. Some hosts and firewalls block AI crawlers by default, without the owner choosing to.
To add an llms.txt, publish a plain text file at yourbusiness.com/llms.txt: a # line with your business name, a > line with one sentence about what you do and where, then ## sections listing your key pages as links.
Read the guide: llms.txt and AI crawler access: a technical guide
Want the whole site checked?
This tool checks one thing. The free audit crawls your whole site the way AI crawlers do, checks your listings too, and emails you a score out of 100 with the three findings that matter most.