Skip to main content
AI Crawler Analytics (labeled Bot Traffic in the platform) lets you see exactly which AI bots and crawlers are visiting your website, how often they visit, and which pages they’re crawling. This gives you a direct signal of how AI engines interact with your content — complementing the response-based metrics like Brand Visibility and Citation Rate.
AI Crawler Analytics requires installing a lightweight tracking script on your website. No data will appear until the script is live.

​
Why it matters

AI search engines like ChatGPT, Perplexity, and Claude crawl your website to fetch content for their responses. Tracking this traffic helps you:
  • Confirm your content is being crawled — if AI bots aren’t visiting, they can’t cite you
  • Identify which pages attract AI attention — double down on content that bots frequently visit
  • Detect crawl pattern changes — a drop in AI crawler visits may foreshadow a drop in visibility
  • Separate AI traffic from human traffic — understand your true audience mix

​
Setup

1

Navigate to AI Crawler Analytics

Open the AI Crawler Analytics page from the sidebar in the Superlines dashboard.
2

Copy the tracking script

The page displays a tracking script unique to your organization. Click Copy to copy the snippet.
3

Add the script to your website

Paste the script into the <body> of every page you want to track. It works with any website — static HTML, WordPress, Next.js, Webflow, or any other platform.
<!-- Superlines Bot Analytics -->
<script>
(function(){
  var img = new Image(1,1);
  var params = "tid=YOUR_TRACKING_ID&url=" + encodeURIComponent(window.location.href);
  if (document.referrer) params += "&ref=" + encodeURIComponent(document.referrer);
  img.src = "https://tracking.superlines.io?" + params;
  img.style.cssText = "position:absolute;left:-9999px;visibility:hidden";
  img.alt = "";
  document.body.appendChild(img);
})();
</script>
<noscript>
  <img src="https://tracking.superlines.io?tid=YOUR_TRACKING_ID"
       width="1" height="1"
       style="position:absolute;left:-9999px;visibility:hidden" alt="">
</noscript>
4

Verify installation

After deploying, return to the AI Crawler Analytics page. The status indicator will show Active once the first visits are recorded — typically within minutes.
The tracking script is a 1×1 transparent pixel — it has zero impact on page load performance and does not use cookies or collect personal data from your visitors.

​
How bot detection works

Superlines identifies bots using two methods:
MethodHow it worksConfidence
User-Agent matchingCompares the visitor’s User-Agent string against a database of 80+ known botsHigh
IP attributionMaps the visitor’s IP address to known LLM provider IP ranges (OpenAI, Anthropic, Perplexity, Mistral)High
IP attribution is especially valuable because some AI engines use headless browsers that don’t identify themselves in the User-Agent string. By matching their IP ranges, Superlines can still attribute the visit to the correct engine.

​
Bot categories

Every detected bot is classified into one of these categories:
CategoryWhat it includesExamples
AI TrainingCrawlers that collect data for training AI modelsGPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider
AI SearchBots that fetch content for real-time AI search responsesOAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User
Search EngineTraditional search engine crawlersGooglebot, Bingbot, Applebot, DuckDuckBot
Social MediaLink preview bots from social platformsFacebookBot, Twitterbot, LinkedInBot, Slackbot
SEO ToolCrawlers from SEO analysis platformsAhrefsBot, SemrushBot, MJ12bot, DotBot
MonitoringUptime and performance monitoring botsUptimeRobot, Pingdom, GTmetrix
Use the category filter on the dashboard to focus on specific bot types like AI Training and AI Search crawlers — the ones most relevant to your AI search visibility.

​
Dashboard overview

The AI Crawler Analytics dashboard provides several views:

​
Summary cards

At the top, you’ll see high-level stats for the selected date range:
  • Top Crawler — the most active bot visiting your site
  • Top URL — the page receiving the most bot traffic
  • Top Category — the most common bot category
  • Top Country — where most bot traffic originates

​
Trend chart

A daily time-series showing bot visit volume over time. You can toggle between:
  • All bots — combined view
  • Per-bot breakdown — individual trend lines for each bot (color-coded by engine)

​
Top bots

A ranked table of the most active bots, showing visit count and percentage of total bot traffic. Each bot displays its category badge and organization.

​
Category breakdown

A pie chart showing the distribution of bot traffic across categories (AI Training, AI Search, Search Engine, etc.).

​
Top pages

Which pages on your site are most frequently crawled by bots. This helps you understand what content AI engines find most valuable.

​
Geographic distribution

Where bot traffic originates from, broken down by country. Most AI engine traffic comes from US and EU data centers, but this can reveal unexpected patterns.

​
Filtering

The dashboard supports several filters to narrow down your analysis:
FilterOptions
Date rangeCustom date picker, presets for 7d / 30d / 90d
Bot filterFilter to a specific bot (e.g., only GPTBot)
Category filterFilter to a specific category (e.g., only AI Search)
Country filterFilter by geographic origin
All botsToggle between showing all bots or filtering to specific categories

​
Human visitor analytics

The dashboard also provides analytics for human visitors:
  • Top referrers — where your human traffic comes from
  • Browsers — browser distribution (Chrome, Safari, Firefox, etc.)
  • Devices — mobile vs. desktop vs. tablet breakdown
This gives you a complete picture of all traffic hitting your tracked pages.

​
Tracking scope

Tracking IDs can be scoped at two levels:
LevelUse case
OrganizationOne tracking ID shared across all your brands. Use this when all brands share the same website domain.
Domain (brand)A separate tracking ID per brand. Use this when your brands have different website domains.
If multiple brands share the same website domain, Superlines automatically shares the tracking ID between them so you don’t need to install the script multiple times.

​
Data and privacy

  • The tracking pixel does not use cookies
  • No personally identifiable information (PII) is collected from human visitors
  • IP addresses are used only for bot attribution and geo-location, not stored long-term for human visitors
  • The script has zero impact on Core Web Vitals and page performance
  • Data is stored in BigQuery with daily partitioning for efficient querying

​
FAQ

Data appears within minutes of the tracking script being live. However, AI bot crawl frequency varies — some bots visit daily, others weekly. Give it 24–48 hours for a representative sample.
No. The tracking script loads a 1×1 transparent GIF asynchronously. It does not block page rendering, execute third-party JavaScript, or impact Core Web Vitals.
Superlines only tracks bot visits — it does not block them. To block specific bots, use your website’s robots.txt file. The data from Superlines helps you make informed decisions about which bots to allow or disallow.
AI bots may not crawl every website frequently. Ensure the tracking script is installed on your most important content pages. If your site is new or has low domain authority, AI crawlers may visit less often. Check that your robots.txt is not blocking AI bots.
AI Training bots (like GPTBot and ClaudeBot) crawl your site to collect data for training future AI models. AI Search bots (like OAI-SearchBot and PerplexityBot) fetch your content in real-time to generate search answers. AI Search bot visits are a more direct signal of your content being used in AI responses.

Key metrics

Learn about Brand Visibility, SOV, and other core metrics.

Analysis process

Understand how Superlines collects and processes AI search data.