Free tool · no signup
Paste your domain and instantly see which AI bots your robots.txt allows or blocks — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more. If you're blocking the crawlers, ChatGPT and friends can't read, cite, or recommend you.
This tool checks one thing: access. Letting the bots in is step one. AUDXY scores your answer-readiness + extractability across your whole site — the part that decides whether AI actually quotes you.
Run full AUDXY — free teaser →More and more people skip Google and just ask ChatGPT, Claude, Perplexity or Google's AI for a recommendation. Those assistants can only suggest a business they were actually allowed to read — and the gatekeeper is a tiny file on your site called robots.txt (the plain-text rulebook that tells automated visitors where they may go). This free AI Crawler Checker reads that live file for you and shows, bot by bot, whether ChatGPT, Claude, Perplexity and Google can crawl and read your website — or whether you are quietly blocking them.
You paste your domain; it fetches your live robots.txt and reads it the way each AI crawler does. In plain English, it tells you:
robots.txt file exists at all — and if it doesn't, that every crawler is allowed in by default.User-agent: *) — the line that can accidentally block everyone at once.A green "allowed" means that bot can read your site. A red "blocked" means it can't. "Partial" means some pages are blocked but not your whole site.
Every page on your site is read by three very different audiences, and they don't all see the same thing:
robots.txt.robots.txt; if it's blocked, you can vanish from ordinary search results.The painful part: most blocks are accidental — a leftover staging rule, a "block everything" line, or a security plugin that bans visitors it doesn't recognise. You can have a beautiful, fast site and still be invisible to the tools your customers now ask first.
A good result is mostly green: the AI crawlers you care about show "allowed," and if you have no robots.txt at all, that's fine too — no rules means everyone is welcome. A result worth acting on shows red "blocked" pills next to bots you actually want reading you, or a catch-all User-agent: * that's blocking everything. "Partial" usually means a folder (like an admin area) is blocked while the rest of your site is open — often exactly what you want, so check which paths before changing anything.
yourdomain.com/robots.txt in a browser and read it top to bottom — it's short.Disallow: / under User-agent: * blocks your entire site from everyone. That's the most common accidental killer.User-agent: * followed by an empty Disallow:. To block only AI training while keeping search, target the specific bot names instead.Only if you let them in. Each assistant sends its own crawler — GPTBot for ChatGPT, ClaudeBot for Claude, PerplexityBot for Perplexity, Google-Extended for Google's AI — and every one of them checks your robots.txt first. If a rule blocks them, they never read your pages. The checker above reads that live file and tells you, bot by bot, who's allowed in.
robots.txt is a small text file in the root of your site that gives automated visitors their instructions: it names a crawler (the user-agent) and lists the paths it may or may not visit. It's the first thing most well-behaved bots — AI crawlers and Google alike — read before anything else, so one wrong line can hide your whole site from them.
Some sites block AI bots on purpose to protect paywalled or proprietary content, or to stay out of training data. But most blocking is accidental — a blanket Disallow: /, a leftover staging rule, or a security plugin that bans unknown visitors. If you want to show up when people ask ChatGPT, Claude, Perplexity or Google for a recommendation, blocking those crawlers quietly removes you from the answer.
Open your robots.txt at yourdomain.com/robots.txt and remove or narrow any Disallow: / rules under the AI user-agents — or under User-agent: * if that catch-all is the culprit. To allow everything, the simplest safe rule is User-agent: * followed by an empty Disallow:. Save, deploy, then re-run this check — the tool reads the live file, so a correct change shows up immediately.
They're separate. Google's normal search crawler (Googlebot) and its AI crawler (Google-Extended) are different user-agents, so you can allow one and block the other. But a blanket Disallow: / under User-agent: * blocks everything at once — including Googlebot — which can erase you from ordinary search results too. That's why catching an accidental block early matters.
Access is step one, not the finish line. Letting the crawlers in means they can read you; whether they understand and quote you depends on how clearly your pages are written and structured. This tool checks access only. To see how quotable your content is, try the Answer-Readiness Checker or run the free AUDXY audit.
Yes — no signup, no card. Paste a domain and get the result. There's a fair daily limit to keep it free for everyone; if you need unlimited checks across many sites, the full AUDXY audit covers that.
Once the door is open, see what the bots actually do with your pages: How AI Reads Your Page shows you your site through an AI's eyes, the Answer-Readiness Checker scores how quotable each page is, and the llms.txt Generator builds the new guide file that points AI assistants straight to your best content. Browse the full set on the free tools hub.