Crawler directory
A reference for the 53 crawlers Visitor Fast recognises: the user agent each one sends, who runs it, why it reads your pages and how its requests are verified.
Fetchers that open a page when someone asks an AI assistant a question or wants a source cited.
ChatGPT-User
OpenAI
User request
Verified bypublished IP list
ChatGPT-Agent
OpenAI
Agent
Verified bypublished IP list
Claude-User
Anthropic
User request
Verified bypublished IP list
Claude-Web
Anthropic
User request
Verified bypublished IP list
Perplexity-User
Perplexity
User request
Verified bypublished IP list
Google-CloudVertexBot
Vertex AI agents
Verified bypublished IP list
Google-Gemini
User request
Verified bypublished IP list
Meta-ExternalFetcher
Meta
User request
Matched by user agent
Amzn-User
Amazon
User request
Matched by user agent
MistralAI-User
Mistral
User request
Matched by user agent
DuckAssistBot
DuckDuckGo
AI answers
Verified bypublished IP list
Crawlers that discover and refresh pages for search engines and AI answer indexes.
OAI-SearchBot
OpenAI
ChatGPT search
Verified bypublished IP list
Claude-SearchBot
Anthropic
Claude search
Verified bypublished IP list
PerplexityBot
Perplexity
Answer engine
Verified bypublished IP list
Amazonbot
Amazon
Alexa and AI
Verified byreverse DNS
YouBot
You.com
Answer engine
Matched by user agent
PhindBot
Phind
Answer engine
Matched by user agent
PetalBot
Huawei
Search and AI
Matched by user agent
Googlebot
Search index
Verified bypublished IP list
bingbot
Microsoft
Search and Copilot
Verified bypublished IP list
Applebot
Apple
Siri and Spotlight
Verified bypublished IP list
DuckDuckBot
DuckDuckGo
Search index
Verified bypublished IP list
YandexBot
Yandex
Search index
Verified byreverse DNS
Baiduspider
Baidu
Search index
Verified byreverse DNS
Slurp
Yahoo
Search index
Matched by user agent
SeznamBot
Seznam
Search index
Matched by user agent
Qwantbot
Qwant
Search index
Matched by user agent
Bravebot
Brave
Search index
Matched by user agent
Crawlers that collect public pages that may be used to train or research AI models.
GPTBot
OpenAI
Model training
Verified bypublished IP list
ClaudeBot
Anthropic
Model training
Verified bypublished IP list
GoogleOther
Research and AI
Verified bypublished IP list
Applebot-Extended
Apple
Apple Intelligence
Verified bypublished IP list
Meta-ExternalAgent
Meta
Model training
Matched by user agent
FacebookBot
Meta
Model training
Matched by user agent
Bytespider
ByteDance
Model training
Matched by user agent
TikTokSpider
ByteDance
Model training
Matched by user agent
CCBot
Common Crawl
Open dataset
Matched by user agent
cohere-training-data-crawler
Cohere
Model training
Matched by user agent
AI2Bot
Allen Institute
Model training
Matched by user agent
DeepSeekBot
DeepSeek
Model training
Matched by user agent
PanguBot
Huawei
Model training
Matched by user agent
Timpibot
Timpi
Model training
Matched by user agent
ImagesiftBot
Hive
Image dataset
Matched by user agent
omgili
Webz.io
Dataset
Matched by user agent
Kangaroo Bot
Kangaroo LLM
Model training
Matched by user agent
Link previews and other automated agents that read pages for a single purpose.
Diffbot
Diffbot
Knowledge graph
Matched by user agent
xAI-Bot
xAI
Model training
Matched by user agent
facebookexternalhit
Meta
Link preview
Matched by user agent
Twitterbot
X
Link preview
Matched by user agent
LinkedInBot
Link preview
Matched by user agent
Slackbot
Slack
Link preview
Matched by user agent
Discordbot
Discord
Link preview
Matched by user agent
TelegramBot
Telegram
Link preview
Matched by user agent
Visitor Fast records every request from AI assistants, search indexes and training crawlers, checks each one against its published IP addresses and shows the pages they read, including missing ones.