AI Crawler Database

The AI Crawler Database is a living reference table of every known AI bot and crawler user agent string currently active on the web. Each entry includes the exact user agent string, the company behind it, whether it's used for training or real-time retrieval, how to block or allow it in your robots.txt, and when it was first documented. The database is updated monthly to keep pace with new bots as they appear.

More than most site owners realize. Beyond the well-known bots from OpenAI, Google, Anthropic, and Perplexity, there are dozens of AI crawlers operated by companies building AI products, search tools, data aggregation services, and large language models across the globe. The number grows every few months as new AI products launch and existing companies spin up additional crawlers for different purposes. The database tracks all of them in one place so you don't have to piece it together from scattered blog posts and changelog entries.

Every entry in the database includes the exact user agent string (what you'd reference in your robots.txt), the parent company or organization, the bot's primary purpose (model training, live retrieval, research indexing, etc.), the specific robots.txt directive to block or allow it, the date it was first spotted in the wild, and any known behavioral notes - like whether the company has publicly committed to respecting robots.txt directives.

This is the single most important distinction in the database. Training crawlers scrape your content to feed into AI model training datasets - your content becomes embedded in the model's weights, but you receive no attribution or traffic in return. Retrieval crawlers fetch your content in real time when a user asks a question, and the AI engine cites your page as a source with a link. Blocking a training crawler protects your content from uncompensated use. Blocking a retrieval crawler cuts you off from AI-driven referral traffic and citations. These are very different decisions, and the database clearly labels each bot so you can make them separately.

The two tools are designed to work together. The Robots.txt AI Bot Checker analyzes your current robots.txt file and shows you which AI bots you're blocking or allowing. The AI Crawler Database is the reference you use to understand what each of those bots actually does and make informed decisions about which ones you want to allow. Check your current configuration with the Robots.txt tool, then look up any unfamiliar bots in the database to decide whether your rules match your intentions.

Monthly, at minimum. The AI crawler landscape shifts frequently - new bots appear, existing bots change user agent strings, companies launch separate crawlers for different functions, and some bots get deprecated entirely. Each monthly update includes new bots that have been documented, changes to existing entries, and removal of bots that are no longer active. Major changes between scheduled updates (like a major AI company launching a new crawler) are added as they're confirmed.

New bots are identified through a combination of server log analysis, industry documentation, official company announcements, community reports, and monitoring of robots.txt changes across major websites. Each bot is verified against its parent company's public documentation where available. If a bot appears in logs but has no official documentation, it's included in the database with a note indicating its unverified status so you can make your own call on how to handle it.

No. User agent strings are self-reported, and there's no technical requirement that forces a crawler to accurately identify itself. Reputable companies use identifiable user agent strings and document them publicly. Less transparent operators may use generic or misleading user agent strings, making them harder to identify and block individually. The database notes where transparency concerns exist so you can factor that into your decisions.

You can add individual disallow rules for each known AI bot user agent string, and the database provides the exact directives for every entry. Some site owners use a blanket approach with broad wildcard rules, but this carries risk - overly aggressive blocking can inadvertently affect legitimate crawlers or future bots that use similar naming patterns. The more targeted approach is using the specific user agent strings listed in the database so you control exactly what's blocked and what isn't.

It depends on your default posture. If your priority is protecting your content and you'd rather opt in to specific bots, blocking unfamiliar crawlers is the safer approach. If your priority is maximizing AI visibility and you'd rather opt out of specific bots, leaving unfamiliar crawlers unblocked gives you broader coverage. The database helps either way - it gives you the information to move any unfamiliar bot from the "unknown" column into an informed decision.