How do you find out which parts of your site AI crawlers actually read?

Separate the agents before you count anything

The agents behave differently and mean different things, so aggregating them produces a number that says nothing:

• Training or index crawlers collect broadly and on their own schedule. Coverage here is about corpus inclusion.
• User-triggered fetchers fire when someone asks a question in a live session. These correlate with actual demand and are the more commercially interesting signal.
• Third-party visibility trackers sweep on a schedule on behalf of somebody else. They inflate your totals and belong in their own bucket.

Verify user agents against published IP ranges. The string is trivially spoofed and a meaningful share of what claims to be a major AI crawler is not.

The measurement that matters

Take the pages that carry your commercial argument. Take the pages the crawlers actually fetched. The gap between those two sets is the finding. Everything else is a vanity count.

Second-order questions worth answering from the same data: which status codes the agents received rather than which ones a browser receives, how much of the crawl budget goes to parameterised or paginated URLs, whether fetch frequency tracks publication frequency, and whether any template returns a materially different response to a non-JavaScript agent.

The most common finding

On client-side rendered applications, the crawlers arrive, receive a document containing almost no content, and leave. The logs show healthy traffic. The pages are effectively empty to every agent that matters. This has come up repeatedly on React and Next.js builds shipped without server-side rendering, and it is invisible in every tool that renders JavaScript before reporting.

Turning it into something ongoing

A one-off read tells you the current state. What changes decisions is the trend: coverage of your commercial pages over time, response codes by agent, and alerts when a template starts behaving differently. That monitoring can be built as a pipeline rather than repeated manually.