40 of 60: What an AI Crawler Saw on Our Own Site

We read our own website the way a text-based AI crawler does and found 40 of 60 actively promoted pages effectively empty — despite complete Schema.org markup. What that taught us about rendering and AI visibility.

A representative Swiss survey (Brandfinity, Swiss Marketing, HEG Fribourg, PME Magazine and M&K; fieldwork December 2025 to March 2026, 194 marketing and sales decision-makers in Switzerland) put a number on it: 61 percent of Swiss marketing teams already adapt their content for AI-powered search, but only 19 percent do so systematically. We've discussed that 42-point gap between intent and execution here before. This time the finding is closer to home — we measured it on ourselves. The simplest test we had never run on ourselves We build the infrastructure that brings brand content, product data and user intent together for AI agents. What we had never systematically checked was: what does a text-based AI crawler actually see when it reads our own website? Not a human in a browser, not a search engine with full JavaScript rendering — a fetcher that loads a page, discards scripts and formatting, and counts whatever plain text is left. That is how many of the tools language models use to fetch web pages actually operate. We built a small script that mimics exactly that: fetch the HTML, strip <script> and <style> blocks, strip every remaining tag, count the words that remain. The first target was a recipe page and a product page. The result, measured on September 10: zero words. Not thin — zero. Even though both pages carried complete structured data in Schema.org format. Why full markup wasn't enough The cause was mundane once it was visible: our website is a client-rendered single-page application. The actual content — title, description, ingredients, price — only appears once JavaScript runs in the browser and fills an initially empty container. A fetcher that doesn't execute JavaScript sees that container empty. That is exactly what many text-based fetchers do: they load the first HTML document, not the result after execution. The structured data didn't help, because it lives inside the same <script> block that a text-only fetcher discards as non-content. Complete markup and zero visible text aren't a contradiction — they're two layers that a text-based fetcher treats very differently, and only one of them counted for it. For comparison: several other properties in our brand family that render server-side were unaffected by this issue. That confirms it wasn't an inherent trait of AI crawlers, but a property of this one implementation — fixable, not unavoidable. The fix — and how much deeper the problem went The …

Author
Nexbid Engineering
Published
2026-09-17
Topics
Research, Verification, Agent Discovery