On August 13, 2026, Microsoft Clarity added an AI Scrape-to-Referral Ratio card to its Bot Analytics report, putting AI crawl volume and AI referral traffic into one number for the first time in a free tool. Search Engine Land’s Barry Schwartz published a Microsoft-supplied screenshot in which one operator sent 41 referral visits against a crawl volume working out to roughly 6,000 scraped pages per visit.
That ratio is the number catalog-heavy Michigan sites have argued about without evidence for two years. It is now measurable in an afternoon, free — and easier to misread than almost any metric shipped this year.
What is Microsoft Clarity’s AI Scrape-to-Referral Ratio?
It is a card inside Clarity’s Bot Analytics report that divides how much of your content an AI operator crawled by how many visits it sent back. Microsoft describes it as a way to “evaluate the tradeoff between content extraction and traffic return,” paired with a ranked operator-level breakdown showing which AI sources return traffic and which only take.
Mapped-domain coverage restricts the calculation to domains where Clarity and your CDN log the same properties, so a partially instrumented subdomain cannot produce a wildly wrong rollup, and Clarity flags coverage gaps rather than quietly averaging. Each referral source also links into the Recordings tab with the filter pre-applied.
That last piece is what makes the setup worth it. Counting AI referrals has been possible in GA4 for a year with regex. Watching thirty of them bounce or hit a quote form has not.
Why did Microsoft build a ratio instead of just counting AI referrals?
Because a raw referral count has no denominator, and the denominator is where the cost sits: bandwidth, origin CPU, and — on a metered CDN — an invoice. AI referrals can climb every month while you lose ground, if crawl volume climbs faster.
The mechanism predicts cases the announcement does not cover. Crawl volume scales with your URL count; referrals scale with query demand. A 40-page HVAC site in Royal Oak and a 40,000-SKU parts catalog in Warren can earn identical AI referrals per month, and the catalog will post a ratio a thousand times worse purely because it has a thousand times more pages to scrape. Read the ratio as a property of your architecture first, content quality second — and treat cross-company comparison as meaningless unless URL counts are within an order of magnitude.
Does a bad ratio mean you should block the crawler?
Usually not, and this is where the tool’s framing deserves pushback. Microsoft positions the metric as input to a keep-or-block decision, but the ratio counts only referral clicks — the least common way AI exposure turns into revenue.
Consider a maintenance manager asking an assistant which supplier carries a discontinued hydraulic fitting. If your spec page is in the model’s context and your company gets named, the manager frequently does not click the citation — they search your brand name, type your domain, or call. Those sessions land as direct or organic, never as an AI referral. The scrape that produced the mention sits in the denominator; the revenue it produced sits where the ratio cannot see it. Block the crawler because the ratio looked ugly and you remove the mention along with the crawl cost.
The defensible use of a bad ratio is narrower: rate-limit an operator hammering paginated or faceted URLs, and exclude low-value URL patterns from crawling. It does not justify blanket-blocking a major assistant unless the bandwidth bill is material — which, for most sites we work with at Rosh Media, it is not.
What did the August 13 studies actually say about AI traffic?
Two datasets landed the same day and point in opposite directions until you look at who was measured.
A Scrunch study reported by Search Engine Land found AI referrals accounted for just 1.1% of visits to news publishers following AI conversations. Scrunch also found readers were 20.5 percentage points more likely to visit a news publisher in the week after a news-related AI conversation — while cautioning that this does not show AI created new demand, only correlation. That caveat is doing heavy lifting and most coverage dropped it.
Shopify, the same day, reported AI-referred sessions to merchant storefronts up 197% year over year in Q2, with organic search up 12% and still sending more traffic than all tracked AI platforms combined. AI-referred shoppers converted at roughly twice the rate of organic visitors in specification-heavy categories, brought in about 1.3 times more first-time customers, and converted at twice the rate again when AI systems used structured Shopify Catalog data rather than scraped or third-party feeds.
Treat the 197% as arithmetic on a small base, published by a vendor selling into the channel it measured. The structured-versus-scraped conversion gap, though, is a mechanism rather than a headline — and the most useful number published this month.
How should Michigan suppliers and aftermarket catalogs read this?
Michigan’s commercial web is unusually exposed, because the state’s economy runs on exactly the content type language models consume most and cite least: dense, tabular, specification-bearing pages.
A fitment catalog in Sterling Heights with year-make-model tables, a Tier 2 supplier in Livonia publishing tolerance sheets, a marine dealer in Grand Rapids listing engine compatibility — prime scrape targets with enormous URL counts, so they post the worst ratios in any Clarity report. They also sit, per Shopify, in the research-intensive categories where AI-referred buyers convert at double the rate. The ugliest ratio in the metro may belong to the site getting the most from AI exposure.
The loss is concrete: a Rochester Hills distributor sees a CDN spike, blocks two AI user agents at the edge, and six months later the quote requests that used to arrive with “I found you through ChatGPT” have stopped — with no report anywhere connecting the two.
How do you set this up in one afternoon?
- Install or confirm the Clarity tag on every domain and subdomain you own, including separate parts or dealer-locator hosts. Partial coverage causes Clarity to caveat or suppress the ratio.
- Open Bot Analytics and record a dated baseline: total AI scrape volume, referrals by operator, and the ratio. Nothing else is worth doing without a before-number.
- Use the operator-level breakdown to separate assistants that return traffic from those that do not, then treat the two groups differently rather than setting one crawler policy for all.
- Click into Recordings for your top AI referral source and watch fifteen to twenty sessions. One question: do these visitors land on a page that answers what sent them, or on a category page they must re-navigate?
- Pull server or CDN logs for the same window and find which URL patterns absorb the crawl. If faceted or paginated URLs dominate, fix that in robots.txt and canonicals — crawl waste under any search model.
- Publish structured product or service data instead of letting third parties describe you. Shopify’s two-times conversion gap between structured catalog data and scraped feeds is the clearest available return, and complete Product, Offer, and FAQPage schema is its shippable version.
- Add a “how did you hear about us” field with an explicit AI assistant option to quote and contact forms. It is the only instrument you own that catches the unattributed path, and it is free.
- Re-check in 30 days against baseline. Direction over 90 days is the signal; one month’s absolute value is not.
What dates should be on the calendar between now and October?
- August 17, 2026 — Google Ads changes how target-based bid strategies behave on budget-limited campaigns; campaigns that had been outperforming their targets will drift toward the configured target. If AI referral traffic has been quietly flattering your blended CPA, this is the week it becomes visible.
- September 12, 2026 — 30 days from a baseline set this week. First honest read on ratio direction.
- Late September — schema and structured-data work should land before crawlers build the index that answers October and November shopping questions, not during it.
- Mid-November 2026 — 90 days of ratio history, the shortest window on which a crawler-policy decision is defensible.
Frequently asked questions
Is Microsoft Clarity free to use for this?
Yes. Clarity is a free analytics product and Bot Analytics with the new AI Scrape-to-Referral Ratio card is included. The cost is setup time and the discipline to instrument every domain you own, since incomplete coverage causes Clarity to caveat or suppress the ratio rather than report a misleading rollup.
What counts as a good Scrape-to-Referral Ratio?
There is no published benchmark, and any number circulating as one deserves skepticism. The ratio scales with URL count, so a large catalog will always look worse than a small service site regardless of content quality. Compare only against your own prior months, and judge direction over at least 90 days.
Should we block AI crawlers to cut bandwidth costs?
Only if the bandwidth cost is material and you have measured the unattributed path first. Blocking removes the mentions that drive branded search and direct visits along with the crawl. Rate-limiting an aggressive operator and excluding faceted URL patterns gets most of the cost relief with far less downside than a blanket block.
Does this replace GA4 for measuring AI traffic?
No. GA4 still owns conversion and revenue attribution. Clarity adds the crawl-side denominator GA4 has never had, plus session recordings filtered to a specific AI source. Run both: GA4 for what AI visitors are worth, Clarity for what they cost you to acquire and how they behave on arrival.
The judgment call
With 68.01% of U.S. Google searches ending without a click in the first four months of 2026 — up from 60.45% in 2024, per SparkToro’s analysis of Similarweb clickstream data — the reflex is to act on whatever new number appears. Clarity’s ratio is a real improvement over guessing, and it is free. It also measures the cheapest form of AI value while ignoring the expensive kind.
Use it to find crawl waste and to rank operators against each other. Do not use it alone to decide who gets to read your site. Separating crawl cost from crawl value in your own logs is the kind of work we do at Rosh Media — starting with the baseline, not the block list.
Comments are closed