Free tools · AI visibility · no sign-up

AI bot blocker generator

Pick the AI bots to block; get the robots.txt block, a Cloudflare WAF expression, nginx, Caddy and .htaccess rules ready to paste.

Featured

Full list (+185 · ai.robots.txt, 2026-11-06)

Which one to use

robots.txt: a request; documented bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) honor it. Not enough for the ones that do not.
Cloudflare WAF: Security → WAF → Custom rules → Create rule → Edit expression; action Block. One expression can hold at most 4 KB; long lists are split for you. Easier: AI Crawl Control (Bot Management) with one switch.
nginx / Caddy / .htaccess: enforced on the server; a User-Agent can be faked, but registered bots use their own name.

Bot list: ai.robots.txt (MIT). The class (training / search / user fetch) is derived from the list's 'function' field; drop any bot you are not sure about.

Other tools (15)

AI visibility

llms.txt generator

Describe your site once; get a valid llms.txt to put at your root so AI assistants know what you are and where to look.

AI crawler check

Enter a domain; see which of 200+ AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended…) your robots.txt allows or blocks, and if you have an llms.txt.

How AI sees your page

Enter a URL; see the page as an AI assistant reads it: clean Markdown, heading tree, word count, links and images without alt text.

SEO

Meta tag & social preview

Enter a URL; check title, description, canonical, robots, hreflang and Open Graph, and preview the Google, X, LinkedIn and WhatsApp cards.

Structured data check

Enter a URL; every JSON-LD block is parsed, flattened and checked for the required and recommended fields of its @type.

Redirect chain check

Enter a URL; follow every hop, catch loops, 302s that should be 301s, missing http→https and www inconsistencies.

robots.txt & sitemap check

Enter a domain; robots.txt syntax is checked and every sitemap is fetched: URL counts, lastmod coverage, size limits and broken XML.

IndexNow key check

Enter a domain and key; we fetch the key file the way Bing does and tell you exactly what is wrong if it fails. Or generate a new key.

SERP snippet preview

Type a title, description and URL; see the Google desktop and mobile snippet with pixel widths and truncation warnings.

Security

Security headers check

Enter a domain; HTTPS redirect, HSTS, CSP, clickjacking, nosniff, referrer policy, exposed .env/.git files, security.txt, SPF and DMARC, each with a fix.

DNS & email security check

Enter a domain; A/AAAA/CNAME/NS/MX/TXT/CAA records plus SPF, DMARC, MTA-STS, DKIM and DNSSEC evaluated with fixes.

Infrastructure

HTTP response check

Enter a URL; status, response time, server, Cloudflare and cache status, compression, HSTS and HTTP/3 in one table with plain-language notes.

Page weight analyzer

Enter a URL; count scripts, styles, images and preloads, find render-blocking resources and measure the first 30 by size.

Domain info (RDAP)

Enter a domain; registrar, creation and expiry dates, days left, status codes explained, name servers and DNSSEC from the registry's RDAP.

Content

Readability analyzer

Paste text; words, sentences, syllables, average sentence length, Flesch Reading Ease (EN) or Ateşman (TR), and the most frequent words.

Have this run every week, automatically

The seops AI crawlers module schedules this check for every site you run and tells you when something changes. One site is free.

Empieza gratis · sin tarjeta Abrir la demo en vivo →