JA
ログイン 無料で始める
Free tools · AI visibility · no sign-up

AI crawler check

Enter a domain; see which of 200+ AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended…) your robots.txt allows or blocks, and if you have an llms.txt.

…or paste your robots.txt

How to read it

  • Allowed: the root is not blocked for this crawler.
  • Partial: some paths are blocked; the root is open.
  • Blocked: Disallow: / applies to this crawler or to the * group.
  • No rule: neither a specific group nor a * group exists; the crawler may fetch everything.
  • Respects: whether the operator documents that it honors robots.txt (ai.robots.txt community list). 'Unclear' does not mean it ignores it.

Blocking training bots (GPTBot, ClaudeBot, Google-Extended) while leaving search bots open (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot) is a common middle path: content stays out of training, you can still be cited in answers. robots.txt is a request; to enforce it use Cloudflare's AI Crawl Control or bot rules.

Generate the blocking rules →

Other tools (15)

AI visibility

llms.txt generator

Describe your site once; get a valid llms.txt to put at your root so AI assistants know what you are and where to look.

AI bot blocker generator

Pick the AI bots to block; get the robots.txt block, a Cloudflare WAF expression, nginx, Caddy and .htaccess rules ready to paste.

How AI sees your page

Enter a URL; see the page as an AI assistant reads it: clean Markdown, heading tree, word count, links and images without alt text.

SEO

Meta tag & social preview

Enter a URL; check title, description, canonical, robots, hreflang and Open Graph, and preview the Google, X, LinkedIn and WhatsApp cards.

Structured data check

Enter a URL; every JSON-LD block is parsed, flattened and checked for the required and recommended fields of its @type.

Redirect chain check

Enter a URL; follow every hop, catch loops, 302s that should be 301s, missing http→https and www inconsistencies.

robots.txt & sitemap check

Enter a domain; robots.txt syntax is checked and every sitemap is fetched: URL counts, lastmod coverage, size limits and broken XML.

IndexNow key check

Enter a domain and key; we fetch the key file the way Bing does and tell you exactly what is wrong if it fails. Or generate a new key.

SERP snippet preview

Type a title, description and URL; see the Google desktop and mobile snippet with pixel widths and truncation warnings.

Security

Security headers check

Enter a domain; HTTPS redirect, HSTS, CSP, clickjacking, nosniff, referrer policy, exposed .env/.git files, security.txt, SPF and DMARC, each with a fix.

DNS & email security check

Enter a domain; A/AAAA/CNAME/NS/MX/TXT/CAA records plus SPF, DMARC, MTA-STS, DKIM and DNSSEC evaluated with fixes.

Infrastructure

HTTP response check

Enter a URL; status, response time, server, Cloudflare and cache status, compression, HSTS and HTTP/3 in one table with plain-language notes.

Page weight analyzer

Enter a URL; count scripts, styles, images and preloads, find render-blocking resources and measure the first 30 by size.

Domain info (RDAP)

Enter a domain; registrar, creation and expiry dates, days left, status codes explained, name servers and DNSSEC from the registry's RDAP.

Content

Readability analyzer

Paste text; words, sentences, syllables, average sentence length, Flesch Reading Ease (EN) or Ateşman (TR), and the most frequent words.

Have this run every week, automatically

The seops AI crawlers module schedules this check for every site you run and tells you when something changes. One site is free.

無料で始める · カード不要 ライブデモを開く →

このページをデモで開く →