AI Crawler Analytics

AI Crawler Analytics (labeled Bot Traffic in the platform) lets you see exactly which AI bots and crawlers are visiting your website, how often they visit, and which pages they’re crawling. This gives you a direct signal of how AI engines interact with your content — complementing the response-based metrics like Brand Visibility and Citation Rate.

AI Crawler Analytics requires installing a lightweight tracking script on your website. No data will appear until the script is live.

Why it matters

AI search engines like ChatGPT, Perplexity, and Claude crawl your website to fetch content for their responses. Tracking this traffic helps you:

Confirm your content is being crawled — if AI bots aren’t visiting, they can’t cite you
Identify which pages attract AI attention — double down on content that bots frequently visit
Detect crawl pattern changes — a drop in AI crawler visits may foreshadow a drop in visibility
Separate AI traffic from human traffic — understand your true audience mix

Setup

Navigate to AI Crawler Analytics

Open the AI Crawler Analytics page from the sidebar in the Superlines dashboard.

Copy the tracking script

The page displays a tracking script unique to your organization. Click Copy to copy the snippet.

Add the script to your website

Paste the script into the <body> of every page you want to track. It works with any website — static HTML, WordPress, Next.js, Webflow, or any other platform.

<!-- Superlines Bot Analytics -->
<script>
(function(){
  var img = new Image(1,1);
  var params = "tid=YOUR_TRACKING_ID&url=" + encodeURIComponent(window.location.href);
  if (document.referrer) params += "&ref=" + encodeURIComponent(document.referrer);
  img.src = "https://tracking.superlines.io?" + params;
  img.style.cssText = "position:absolute;left:-9999px;visibility:hidden";
  img.alt = "";
  document.body.appendChild(img);
})();
</script>
<noscript>
  <img src="https://tracking.superlines.io?tid=YOUR_TRACKING_ID"
       width="1" height="1"
       style="position:absolute;left:-9999px;visibility:hidden" alt="">
</noscript>

Verify installation

After deploying, return to the AI Crawler Analytics page. The status indicator will show Active once the first visits are recorded — typically within minutes.

The tracking script is a 1×1 transparent pixel — it has zero impact on page load performance and does not use cookies or collect personal data from your visitors.

How bot detection works

Superlines identifies bots using two methods:

Method	How it works	Confidence
User-Agent matching	Compares the visitor’s User-Agent string against a database of 80+ known bots	High
IP attribution	Maps the visitor’s IP address to known LLM provider IP ranges (OpenAI, Anthropic, Perplexity, Mistral)	High

IP attribution is especially valuable because some AI engines use headless browsers that don’t identify themselves in the User-Agent string. By matching their IP ranges, Superlines can still attribute the visit to the correct engine.

Bot categories

Every detected bot is classified into one of these categories:

Category	What it includes	Examples
AI Training	Crawlers that collect data for training AI models	GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider
AI Search	Bots that fetch content for real-time AI search responses	OAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User
Search Engine	Traditional search engine crawlers	Googlebot, Bingbot, Applebot, DuckDuckBot
Social Media	Link preview bots from social platforms	FacebookBot, Twitterbot, LinkedInBot, Slackbot
SEO Tool	Crawlers from SEO analysis platforms	AhrefsBot, SemrushBot, MJ12bot, DotBot
Monitoring	Uptime and performance monitoring bots	UptimeRobot, Pingdom, GTmetrix

Use the category filter on the dashboard to focus on specific bot types like AI Training and AI Search crawlers — the ones most relevant to your AI search visibility.

Dashboard overview

The AI Crawler Analytics dashboard provides several views:

Summary cards

At the top, you’ll see high-level stats for the selected date range:

Top Crawler — the most active bot visiting your site
Top URL — the page receiving the most bot traffic
Top Category — the most common bot category
Top Country — where most bot traffic originates

Trend chart

A daily time-series showing bot visit volume over time. You can toggle between:

All bots — combined view
Per-bot breakdown — individual trend lines for each bot (color-coded by engine)

Top bots

A ranked table of the most active bots, showing visit count and percentage of total bot traffic. Each bot displays its category badge and organization.

Category breakdown

A pie chart showing the distribution of bot traffic across categories (AI Training, AI Search, Search Engine, etc.).

Top pages

Which pages on your site are most frequently crawled by bots. This helps you understand what content AI engines find most valuable.

Geographic distribution

Where bot traffic originates from, broken down by country. Most AI engine traffic comes from US and EU data centers, but this can reveal unexpected patterns.

Filtering

The dashboard supports several filters to narrow down your analysis:

Filter	Options
Date range	Custom date picker, presets for 7d / 30d / 90d
Bot filter	Filter to a specific bot (e.g., only GPTBot)
Category filter	Filter to a specific category (e.g., only AI Search)
Country filter	Filter by geographic origin
All bots	Toggle between showing all bots or filtering to specific categories

Human visitor analytics

The dashboard also provides analytics for human visitors:

Top referrers — where your human traffic comes from
Browsers — browser distribution (Chrome, Safari, Firefox, etc.)
Devices — mobile vs. desktop vs. tablet breakdown

This gives you a complete picture of all traffic hitting your tracked pages.

Tracking scope

Tracking IDs can be scoped at two levels:

Level	Use case
Organization	One tracking ID shared across all your brands. Use this when all brands share the same website domain.
Domain (brand)	A separate tracking ID per brand. Use this when your brands have different website domains.

If multiple brands share the same website domain, Superlines automatically shares the tracking ID between them so you don’t need to install the script multiple times.

Data and privacy

The tracking pixel does not use cookies
No personally identifiable information (PII) is collected from human visitors
IP addresses are used only for bot attribution and geo-location, not stored long-term for human visitors
The script has zero impact on Core Web Vitals and page performance
Data is stored in BigQuery with daily partitioning for efficient querying

FAQ

How long until I see data?

Data appears within minutes of the tracking script being live. However, AI bot crawl frequency varies — some bots visit daily, others weekly. Give it 24–48 hours for a representative sample.

Does this affect my website performance?

No. The tracking script loads a 1×1 transparent GIF asynchronously. It does not block page rendering, execute third-party JavaScript, or impact Core Web Vitals.

Can I block specific bots using this data?

Superlines only tracks bot visits — it does not block them. To block specific bots, use your website’s robots.txt file. The data from Superlines helps you make informed decisions about which bots to allow or disallow.

Why don't I see any AI bot visits?

AI bots may not crawl every website frequently. Ensure the tracking script is installed on your most important content pages. If your site is new or has low domain authority, AI crawlers may visit less often. Check that your robots.txt is not blocking AI bots.

What's the difference between AI Training and AI Search bots?

AI Training bots (like GPTBot and ClaudeBot) crawl your site to collect data for training future AI models. AI Search bots (like OAI-SearchBot and PerplexityBot) fetch your content in real-time to generate search answers. AI Search bot visits are a more direct signal of your content being used in AI responses.

Key metrics

Learn about Brand Visibility, SOV, and other core metrics.

Analysis process

Understand how Superlines collects and processes AI search data.

Getting Started

Analytics

Features

Tools

Content

Website

Prompts

Account Management

Metrics & Data

AI Crawler Analytics

Why it matters

Setup

How bot detection works

Bot categories

Dashboard overview

Summary cards

Trend chart

Top bots

Category breakdown

Top pages

Geographic distribution

Filtering

Human visitor analytics

Tracking scope

Data and privacy

FAQ

Key metrics

Analysis process

​Why it matters

​Setup

​How bot detection works

​Bot categories

​Dashboard overview

​Summary cards

​Trend chart

​Top bots

​Category breakdown

​Top pages

​Geographic distribution

​Filtering

​Human visitor analytics

​Tracking scope

​Data and privacy

​FAQ

Key metrics

Analysis process

Why it matters

Setup

How bot detection works

Bot categories

Dashboard overview

Summary cards

Trend chart

Top bots

Category breakdown

Top pages

Geographic distribution

Filtering

Human visitor analytics

Tracking scope

Data and privacy

FAQ