Tüm liste (+185 · ai.robots.txt, 2026-11-06) AddSearchBot AI Search Crawlers AgentDataBot AI Data Providers AgentTimes The Agent Times · Data Scraper from RSS Feeds AGI Agent AGI, Inc · AI Agents AI2Bot Ai2 · Content is used to train open language models AI2Bot-DeepResearchEval Ai2, a non-profit AI research institute · AI Assistants Ai2Bot-Dolma Ai2 · Content is used to train open language models aiHitBot aiHit · A massive, artificial intelligence/machine learning, automated system AIWebIndex Lyrenth · AI Search Crawlers amazon-kendra Amazon · Collects data for AI natural language search amazon-QBusiness AI Assistants AmazonBuyForMe Amazon · AI Agents Amzn-SearchBot AI Search Crawlers Amzn-User Amazon, used for fetching web content to answer user queries through Alexa and other Amazon AI services · AI Assistants Andibot Andi · Search engine using generative AI, AI Search Assistant Anomura Direqt · Collects data for AI search anthropic-ai Anthropic · Scrapes data to train Anthropic's AI products ApifyBot AI Data Providers ApifyWebsiteContentCrawler AI Data Providers Applebot AI Search Crawlers Aranet-SearchBot Undocumented AI Agents atlassian-bot Atlassian · AI search, assistants and agents Awario Awario · AI Data Scrapers AzureAI-SearchBot AI Search Crawlers bedrockbot Amazon · Data scraping for custom AI applications bigsur.ai Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for busi… · AI Assistants BixelBot AI Data Providers Bravebot https://safe.search.brave.com/help/brave-search-crawler · AI Data Providers Brightbot AI Data Providers Brightbot 1.0 https://brightdata.com/brightbot · LLM/AI training BuddyBot BuddyBotLearning · AI Learning Companion CartAI Agentic Commerce CartAI that accesses online stores to support user-directed shopping and checkout workflows · AI Agents Channel3Bot AI Search Crawlers ChatGLM-Spider AI Data Scrapers ChatGPT Agent OpenAI · AI Agents Claude-Agent AI Agents Claude-Code AI Coding Agents Claude-Web Anthropic · Undocumented AI Agents CledaraBot Cledara that visits SaaS providers to complete customer-approved administrative tasks, including invoice collection and user managem… · AI Agents Cloudflare-AutoRAG Cloudflare · Collects data for AI search CloudflareBrowserRenderingCrawler Cloudflare that returns rendered website content for research, monitoring, and AI data workflows · AI Data Providers CloudVertexBot AI Data Scrapers Code AI Coding Agents Codex AI Coding Agents cohere-ai Cohere · Retrieves data to provide responses to user-initiated prompts cohere-training-data-crawler Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products · AI Data Scrapers Cotoyogi ROIS · AI LLM Scraper CragCrawler CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services · AI Data Providers Crawl4AI Undocumented AI Agents Crawlspace Crawlspace · AI Data Providers Cursor AI Coding Agents Datenbank Crawler Datenbank · AI Data Scrapers DeepSeekBot DeepSeek · Training language models and improving AI products Devin Devin AI · AI Coding Agents Diffbot Diffbot · AI Data Providers Diffbot-User Diffbot · AI Assistants Dot AI Agents DoubaoBot ByteDance · AI crawler for ByteDance's Doubao AI assistant Echobot Bot Echobox · AI Data Scrapers EchoboxBot Echobox · Data collection to support AI-powered products ElevenlabsBot ElevenLabs · AI Assistants ElevenlabsCrawler ElevenLabs · AI Data Scraper ERNIEBot Baidu · Collects public web content for Baidu's ERNIE large language models ExaBot AI Data Providers ExaSearchBot Exa · AI Search Crawlers ExaUser AI Assistants FacebookBot Meta/Facebook · Training language models facebookexternalhit Meta/Facebook · Ostensibly only for sharing, but likely used as an AI crawler as well Factset_spyderbot Factset · AI model training FirecrawlAgent Firecrawl that extracts web content and converts it into structured data for use in LLM and AI applications · AI Data Providers FriendlyCrawler We are using the data from the crawler to build datasets for machine learning experiments GeistHaus-PageFetcher GeistHaus, a company developing AI systems for therapy and psychological assessment · AI Assistants Gemini-Deep-Research AI Assistants Google-Agent AI Agents Google-CloudVertexBot Google · Build and manage AI models for businesses employing Vertex AI Google-Firebase Google · Used as part of AI apps developed by users of Google's Firebase AI products Google-Gemini-CLI AI Coding Agents Google-NotebookLM AI Assistants GoogleAgent-Mariner Google · AI Agents GoogleAgent-URLContext Google that retrieves web content on behalf of Gemini API users · AI Assistants GoogleOther Google · Scrapes data GoogleOther-Image Google · Scrapes data GoogleOther-Video Google · Scrapes data Grok Bot AI Agents GrokAppAndroid xAI · AI Search Crawlers GrokBot xAI · AI Search Crawlers HarkBot AI Agents HenkBot AI Data Providers Hunyuan Tencent · Hunyuan is a training crawler operated by Tencent iAskBot Undocumented AI Agents iaskspider Undocumented AI Agents iaskspider/2.0 iAsk · Crawls sites to provide answers to user queries ICC-Crawler NICT · Scrapes data to train and support AI technologies ImagesiftBot ImageSift · ImageSiftBot is a web crawler that scrapes the internet for publicly available images to support their suit… imageSpider AI Data Scrapers img2dataset img2dataset · Scrapes images for use in LLMs Instinct AI Agents ISSCyberRiskCrawler ISS-Corporate · Scrapes data to train machine learning models kagi-fetcher Kagi that fetches web content to answer user queries through Kagi AI, their suite of AI-powered tools including Assistant, Res… · AI Assistants Kangaroo Bot AI Data Scrapers KeenableBot Keenable · AI Search Crawlers Kimi-Agent Moonshot AI that browses websites while carrying out user-directed research and other tasks · AI Agents Kimi-SearchBot Moonshot AI · AI Search Crawlers Kimi-User Moonshot AI that fetches web content on behalf of users interacting with Kimi · AI Assistants KimiBot Moonshot AI · AI Data Scrapers KlaviyoAIBot Klaviyo · AI Assistants KunatoCrawler Undocumented AI Agents laion-huggingface-processor AI Data Scrapers LAIONDownloader Large-scale Artificial Intelligence Open Network · AI tools and models for machine learning research LCC AI Data Scrapers Lightpanda Anyone who downloads the Lightpanda client. Possibly being used by a Grok-adjacent organization's botnet. · AI Data Scrapers LinerBot AI Assistants Linguee Bot Linguee · AI powered translation service LinkupBot AI Search Crawlers Manus-User Butterfly Effect, a company based in China · AI Agents meta-externalfetcher AI Assistants meta-webindexer Meta · AI Assistants MistralAI-Index Mistral AI · Indexes web content for Mistral AI's search, used to answer questions in Le Chat. Per Mistral, not used for… MistralAI-Training Mistral AI · AI Data Scrapers MistralAI-User Mistral · AI Assistants MistralAI-User/1.0 Mistral AI · Takes action based on user prompts MoonshotBot Kimi · MoonshotBot is a training crawler Mozilla-Tabstack Mozilla that performs programmatic, AI-driven interactions with web content through Tabstack · AI Data Providers Muse AI Agents MyCentralAIScraperBot AI data scraper NagetBot Naget Inc (founded by Chris Samarinas, headquarter in Amherst, Massachusetts) · AI data scraper nbot NBot · AI Search Crawlers netEstate Imprint Crawler netEstate · AI Data Scrapers newsai AI data scraper NotebookLM AI Assistants NovaAct AI Agents OAI-AdsBot OpenAI · Validates and targets ads on ChatGPT omgili Webz.io · Data is sold omgilibot Webz.io · Data is sold OpenAI OpenAI · Unclear at this time opencode AI Coding Agents Operator AI Agents PanguBot the Chinese company Huawei · AI Data Scrapers Panscient Panscient · Data collection and analysis using machine learning and AI panscient.com Panscient · Data collection and analysis using machine learning and AI Payhawk Invoice Fetching Agent Payhawk that retrieves invoices from supplier websites for users and attaches them to expense records · AI Agents PetalBot Huawei · Used to provide recommendations in Hauwei assistant and AI search services PhindBot phind · AI Assistants Poggio-Citations Poggio, a company that provides AI sales enablement tools for creating tailored narratives, business cases, and account plan… · AI Assistants Poseidon Research Crawler Poseidon Research · AI research crawler qodercli Qoder that helps developers build, debug, and modify software from the terminal · AI Coding Agents QualifiedBot Qualified · AI Assistants Querit-SearchBot Querit that indexes web content for their search API service, which is designed to provide real-time search results for larg… · AI Data Providers QueritBot Querit, a company providing a search API for large language model integration · AI Data Providers QuillBot Quillbot · Company offers AI detection, writing tools and other services quillbot.com Quillbot · Company offers AI detection, writing tools and other services QwenBot Alibaba · Collects public web content for Alibaba's Qwen (Tongyi Qianwen) large language models Reflectionbot Reflection · Undocumented AI Agents SBIntuitionsBot SB Intuitions · Uses data gathered in AI development and information analysis Scrapy Zyte · Scrapes data for a variety of uses including training AI ScryBot Scry · Used with AI agents to scrape data for answers to queries SemrushBot-OCOB Semrush · Crawls your site for ContentShake AI tool SemrushBot-SWA Semrush · Checks URLs on your site for SEO Writing Assistant Shap-User AI Assistants ShapBot Parallel · AI Data Providers Sidetrade indexer bot Sidetrade · Extracts data for a variety of uses including training AI Spider AI Data Scrapers TavilyBot AI Data Providers Terra Cotta AI Data Providers TerraCotta Ceramic AI · AI Data Providers Thinkbot Thinkbot · Insights on AI integration and automation TikTokSpider ByteDance · LLM training Timpibot Timpi · Scrapes data for use in training LLMs TongyiBot Alibaba that fetches web content for the Tongyi Qianwen assistant and related Qwen-generated answers · AI Assistants Trae AI Coding Agents TwinAgent Twin, a platform that creates automated workers to perform tasks by integrating with APIs and controlling web applications through browser automa… · AI Agents UseAI AI Assistants VelenPublicWebCrawler Velen Crawler · Scrapes data for business data sets and machine learning models WARDBot WEBSPARK · AI Data Scrapers Webzio-Extended AI Data Scrapers wpbot QuantumCloud · Live chat support and lead generation WRTNBot Undocumented AI Agents xAI-Grok xAI · xAI-Grok is an xAI crawler token associated with Grok content collection YaK Meltwater · According to the Meltwater Consumer Intelligence page 'By applying AI, data science, and market research ex… YandexAdditional Yandex · Scrapes/analyzes data for the YandexGPT LLM YandexAdditionalBot Yandex · Scrapes/analyzes data for the YandexGPT LLM YiBot 01.ai, an AI company founded by Kai-Fu Lee. · This bot crawls the web to gather data and content for indexation and AI model training YiyanBot Baidu that fetches web content for the yiyan · AI Assistants YouBot You · Scrapes data for search engine and LLMs ZanistaBot AI Search Crawlers