> ## Documentation Index > Fetch the complete documentation index at: https://docs.superlines.io/llms.txt > Use this file to discover all available pages before exploring further. # AI Crawler Analytics > Track and analyze AI bot traffic on your website to understand how AI engines crawl your content. AI Crawler Analytics (labeled **Bot Traffic** in the platform) lets you see exactly which AI bots and crawlers are visiting your website, how often they visit, and which pages they're crawling. This gives you a direct signal of how AI engines interact with your content — complementing the response-based metrics like Brand Visibility and Citation Rate. AI Crawler Analytics requires installing a lightweight tracking script on your website. No data will appear until the script is live. ## Why it matters AI search engines like ChatGPT, Perplexity, and Claude crawl your website to fetch content for their responses. Tracking this traffic helps you: * **Confirm your content is being crawled** — if AI bots aren't visiting, they can't cite you * **Identify which pages attract AI attention** — double down on content that bots frequently visit * **Detect crawl pattern changes** — a drop in AI crawler visits may foreshadow a drop in visibility * **Separate AI traffic from human traffic** — understand your true audience mix ## Setup Open the **AI Crawler Analytics** page from the sidebar in the Superlines dashboard. The page displays a tracking script unique to your organization. Click **Copy** to copy the snippet. Paste the script into the `` of every page you want to track. It works with any website — static HTML, WordPress, Next.js, Webflow, or any other platform. ```html theme={null} ``` After deploying, return to the AI Crawler Analytics page. The status indicator will show **Active** once the first visits are recorded — typically within minutes. The tracking script is a 1×1 transparent pixel — it has zero impact on page load performance and does not use cookies or collect personal data from your visitors. ## How bot detection works Superlines identifies bots using two methods: | Method | How it works | Confidence | | ----------------------- | ------------------------------------------------------------------------------------------------------ | ---------- | | **User-Agent matching** | Compares the visitor's User-Agent string against a database of 80+ known bots | High | | **IP attribution** | Maps the visitor's IP address to known LLM provider IP ranges (OpenAI, Anthropic, Perplexity, Mistral) | High | IP attribution is especially valuable because some AI engines use headless browsers that don't identify themselves in the User-Agent string. By matching their IP ranges, Superlines can still attribute the visit to the correct engine. ## Bot categories Every detected bot is classified into one of these categories: | Category | What it includes | Examples | | ----------------- | --------------------------------------------------------- | ------------------------------------------------------------ | | **AI Training** | Crawlers that collect data for training AI models | GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider | | **AI Search** | Bots that fetch content for real-time AI search responses | OAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User | | **Search Engine** | Traditional search engine crawlers | Googlebot, Bingbot, Applebot, DuckDuckBot | | **Social Media** | Link preview bots from social platforms | FacebookBot, Twitterbot, LinkedInBot, Slackbot | | **SEO Tool** | Crawlers from SEO analysis platforms | AhrefsBot, SemrushBot, MJ12bot, DotBot | | **Monitoring** | Uptime and performance monitoring bots | UptimeRobot, Pingdom, GTmetrix | Use the category filter on the dashboard to focus on specific bot types like AI Training and AI Search crawlers — the ones most relevant to your AI search visibility. ## Dashboard overview The AI Crawler Analytics dashboard provides several views: ### Summary cards At the top, you'll see high-level stats for the selected date range: * **Top Crawler** — the most active bot visiting your site * **Top URL** — the page receiving the most bot traffic * **Top Category** — the most common bot category * **Top Country** — where most bot traffic originates ### Trend chart A daily time-series showing bot visit volume over time. You can toggle between: * **All bots** — combined view * **Per-bot breakdown** — individual trend lines for each bot (color-coded by engine) ### Top bots A ranked table of the most active bots, showing visit count and percentage of total bot traffic. Each bot displays its category badge and organization. ### Category breakdown A pie chart showing the distribution of bot traffic across categories (AI Training, AI Search, Search Engine, etc.). ### Top pages Which pages on your site are most frequently crawled by bots. This helps you understand what content AI engines find most valuable. ### Geographic distribution Where bot traffic originates from, broken down by country. Most AI engine traffic comes from US and EU data centers, but this can reveal unexpected patterns. ## Filtering The dashboard supports several filters to narrow down your analysis: | Filter | Options | | ------------------- | ------------------------------------------------------------------- | | **Date range** | Custom date picker, presets for 7d / 30d / 90d | | **Bot filter** | Filter to a specific bot (e.g., only GPTBot) | | **Category filter** | Filter to a specific category (e.g., only AI Search) | | **Country filter** | Filter by geographic origin | | **All bots** | Toggle between showing all bots or filtering to specific categories | ## Human visitor analytics The dashboard also provides analytics for human visitors: * **Top referrers** — where your human traffic comes from * **Browsers** — browser distribution (Chrome, Safari, Firefox, etc.) * **Devices** — mobile vs. desktop vs. tablet breakdown This gives you a complete picture of all traffic hitting your tracked pages. ## Tracking scope Tracking IDs can be scoped at two levels: | Level | Use case | | ------------------ | ------------------------------------------------------------------------------------------------------ | | **Organization** | One tracking ID shared across all your brands. Use this when all brands share the same website domain. | | **Domain (brand)** | A separate tracking ID per brand. Use this when your brands have different website domains. | If multiple brands share the same website domain, Superlines automatically shares the tracking ID between them so you don't need to install the script multiple times. ## Data and privacy * The tracking pixel does **not** use cookies * No personally identifiable information (PII) is collected from human visitors * IP addresses are used only for bot attribution and geo-location, not stored long-term for human visitors * The script has **zero impact** on Core Web Vitals and page performance * Data is stored in BigQuery with daily partitioning for efficient querying ## FAQ Data appears within minutes of the tracking script being live. However, AI bot crawl frequency varies — some bots visit daily, others weekly. Give it 24–48 hours for a representative sample. No. The tracking script loads a 1×1 transparent GIF asynchronously. It does not block page rendering, execute third-party JavaScript, or impact Core Web Vitals. Superlines only tracks bot visits — it does not block them. To block specific bots, use your website's `robots.txt` file. The data from Superlines helps you make informed decisions about which bots to allow or disallow. AI bots may not crawl every website frequently. Ensure the tracking script is installed on your most important content pages. If your site is new or has low domain authority, AI crawlers may visit less often. Check that your `robots.txt` is not blocking AI bots. **AI Training** bots (like GPTBot and ClaudeBot) crawl your site to collect data for training future AI models. **AI Search** bots (like OAI-SearchBot and PerplexityBot) fetch your content in real-time to generate search answers. AI Search bot visits are a more direct signal of your content being used in AI responses. Learn about Brand Visibility, SOV, and other core metrics. Understand how Superlines collects and processes AI search data.