On-Page Signals That AI Crawlers Look For

AI visibility is not something that comes naturally. No, it is against nature to struggle for visibility to a machine. To strive with others for a machine's attention. To be distinct enough for a machine to listen to what you intend to say.

Yet we must do it. Because the landscape of visibility on the web is shifting fast. So fast that many individuals and businesses do not see it. Visibility has evolved beyond SEO. It is not about what your search engines say about you anymore. It is now about what AI tell or do not tell people about you.

And this is harder.

Once upon a search engine, you fought for attention when all the rules had a roof to stay under. That roof was Google. It published guidelines, its engineers gave talks, and the whole industry argued over the same short list of ranking factors. You could read up on the rules, open your site, and improve the content to line up with them. Some of it was tedious, but it was learnable, and you always knew which game you were playing.

Take backlinks. They were the closest thing the old world had to a public scoreboard. Another site linking to yours counted as a vote, a vote from a trusted site counted for more, and you could literally count them. Tools sprang up to do exactly that. If a competitor outranked you, you could open a report, see that they had two hundred more referring domains than you did, and know where your next month of work was going (and whom you would be sending very polite emails to).

Now try that with an AI model. Which backlink made ChatGPT mention a brand in its answer? Did Claude lean on a page because of who linked to it, or because of how clearly the page was written? Does Perplexity care about the same things Gemini does? Nobody has published that scoreboard, and the answer for one model is often not the answer for the next. Worse, ask the same question twice and you may get two different citations.

Backlinks, they have not gone anywhere. Many AI answers still lean on search indexes, and in those indexes it was links that decided what got found, long before any model began to talk. But the old throne is shared now, and a link, however loud, sits as one voice among many.

The elders had this one covered: Otu osisi anaghị abụ ọhịa. One tree does not make a forest. Your backlinks are the tree, and the forest is everything else a machine is weighing about you, which is a lot of everything.

Which is why the fence in our adage matters, and matters more than it did yesterday. Gone is the day of crowing for one listener on the clock. Today you crow for several, and each of them wakes up tuned a little differently, so that what rings clear for one is only a rumour to the next.

Who Exactly Is on the Clock?

Before we argue about what to crow, let us meet the ones listening. “AI crawler” sounds like a single creature with one ear to the ground, when in truth it is a whole household of bots that wake at different hours, for different reasons, and play by different rules. They are also not small listeners. Over one month on Vercel’s network, OpenAI’s GPTBot made 569M requests and Anthropic’s crawler made 370M, which together came to about 20% of Googlebot’s 4.5B (Zecchini et al., 2024).

The easiest way to tell them apart is to ask what each one is there to do. Most fall into three jobs: gathering content that may help train a model, building the search index an AI answer draws from, and fetching one page because a person just asked for it. The same company often runs all three, and that is where most of the confusion starts.

Company	Bot	What it is there to do
OpenAI	GPTBot	Collects content that may be used for model training
OpenAI	OAI-SearchBot	Surfaces sites in ChatGPT’s search results
OpenAI	ChatGPT-User	Visits a page when a user’s request calls for it
Anthropic	ClaudeBot	Collects content that may be used for model training
Anthropic	Claude-SearchBot	Indexes content to improve Claude’s search results
Anthropic	Claude-User	Visits a page when a user’s question calls for it
Perplexity	PerplexityBot	Surfaces and links sites in Perplexity’s search results
Perplexity	Perplexity-User	Fetches a page for a user action
Google	Googlebot	Crawls for Search, and the Search index also grounds Gemini
Google	Google-Extended	Not a crawler at all (more on this below)

The search bots are the ones you want to be heard by. OpenAI says a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers, though it can still appear as a navigational link (OpenAI, n.d.). Anthropic says that disabling Claude-SearchBot stops your content from being indexed for search and may reduce your visibility (Anthropic, n.d.), and Perplexity recommends allowing PerplexityBot if you want to appear in its results (Perplexity, n.d.). Training bots sit behind a separate door, since each of these bots has its own robots.txt entry (Anthropic, n.d.; OpenAI, n.d.). Welcoming one and turning away another is a choice you make bot by bot.

Then there are the user-triggered fetchers, which behave a little differently. Perplexity says Perplexity-User is not used for web crawling or training, and because a person requested the fetch, it generally ignores robots.txt (Perplexity, n.d.). Anthropic describes Claude-User as the agent that may visit a site when someone asks Claude a question (Anthropic, n.d.). The logic is the same one your own browser lives under, since nobody asks robots.txt for permission before clicking a link.

Google-Extended is the odd one out. Google’s documentation says it has no user agent string of its own, and that crawling is done by Google’s existing agents, with the token used only as a control. It lets you decide whether content Google crawls may be used to train future Gemini models and to ground answers in Gemini Apps and Vertex AI, and it has no effect on your inclusion in Google Search or your ranking there (Google, n.d.). So it works like a sign on the gate rather than a visitor knocking at it.

Here is where the fence in our adage starts to matter. These listeners do not all hear the same way. A joint study by Vercel and MERJ found that none of the major AI crawlers rendered JavaScript, a group that included OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot and PerplexityBot. Gemini, on the other hand, rides on Googlebot’s infrastructure and can render fully, and AppleBot renders too (Zecchini et al., 2024). The OpenAI and Anthropic crawlers did download JavaScript files (in 11.5% and 23.84% of their requests respectively) but never executed them (Zecchini et al., 2024).

A crow that only sounds after your scripts have run is a rooster in the next compound. There is some mercy in the same study, though. Content that arrives in the initial HTML response, such as JSON data or delayed React Server Components, may still be picked up, because AI models can interpret non-HTML content (Zecchini et al., 2024).

Can the Fence Be Seen?

We have spend enough time looking at the stats in the last section. Let's take a brief pause. Ask yourself this question:

Can the fence be seen?

Now internalize it and ask again:

Can your fence be seen? Or can your website's fence be seen?

What is that fence? The fence is everything that stands between a crawler and your words. Most of us only think about the words, so the fence goes unnoticed until nobody can hear us. In practice it has four boards, and like the memory, the ear and the crowd, they are easier to take one at a time.

Let’s start with the gate, which is your robots.txt file.

It is the first thing a well-behaved crawler reads, and Google says its common crawlers always obey it when crawling automatically (Google, n.d.). Anthropic says its bots honour it too (Anthropic, n.d.). The catch is that each bot reads its own line. OAI-SearchBot, GPTBot, ClaudeBot, Claude-SearchBot and PerplexityBot are separate names, so a welcome written for one does nothing for the others. An old robots.txt written before AI crawlers existed can also shut them all out through a single catch-all disallow rule (Vercel, n.d.).

So, go to your site's homepage right now, then add /robots.xml at the end of the URL and press enter. For example, this site's homepage URL lives at unclaimedbyai.com. If I appended /robots.xml to the end, it becomes:

https://www.unclaimedbyai.com/robots.txt

And when I open it on the browser, I get:

User-Agent: *
Allow: /api/docs
Allow: /api/docs.md
Allow: /api/blog/images/
Disallow: /auth
Disallow: /dashboard
Disallow: /admin
Disallow: /api/v1
Disallow: /api/admin
Disallow: /api/blog/boost-state
Disallow: /api/blog/company-meta

Sitemap: https://www.unclaimedbyai.com/sitemap.xml

Now, check yours! Try reading it the way a bot would read it. Then come back and read me read this site's robots.txt.

Now let’s read it the way a bot does, because a crawler does not skim. It first looks for the group that carries its own name, and when it finds none, it falls back to the group marked with an asterisk (Koster et al., 2022). Our file has only that one group, so GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot all walk away with exactly the same instructions. (Perplexity-User would barely glance at it, since Perplexity says it generally ignores robots.txt for requests a person triggers [Perplexity, n.d.].)

From there, the bot tests each URL against the rules. When more than one rule matches, the most specific one wins, which in practice means the longest path. A URL that matches no rule at all is allowed (Koster et al., 2022). That last part surprises people, because in robots.txt silence counts as permission. Nothing in our file mentions the blog, the product page or the pricing page, so every bot above is free to read them.

URL	What the bot decides
/blog	Allowed, because no rule matches
/pricing	Allowed, because no rule matches
/api/docs	Allowed, by an explicit Allow line
/api/blog/images/	Allowed, by an explicit Allow line
/dashboard	Blocked
/api/v1	Blocked

The Disallow lines are the private rooms: sign-in, dashboard, admin and a handful of internal API routes that no crawler has any business reading. The Allow lines are the more interesting ones. Strictly speaking they are not even needed, since nothing blocks those paths in the first place. Still, they earn their place. They say out loud that the docs and a markdown copy of them are meant to be read, and they protect /api/blog/images/, where our blog thumbnails live. If a broad Disallow: /api ever gets added, the longer Allow paths would still win and the images would stay visible. The last line, the Sitemap, sits outside any group and hands every visitor the address of sitemap.xml, which is the list of pages we want found.

So the gate says come in. But a file that says yes is only as good as the door standing in front of it.

Then comes the guard.

A crawler can be welcomed by your robots.txt and still be turned away before it ever reads the file. Your host, your CDN or your firewall may be standing at the door with its own list. Cloudflare began blocking known AI crawlers by default for new domains in July 2025 ("Cloudflare Hits the Kill Switch," 2025). In July 2026 it replaced that single switch with three categories (search, training and agent), and from 15 September 2026 its new-domain defaults block training and agent bots on pages that carry ads while search stays allowed ("Cloudflare Gives Website Owners," 2026). That is a fair choice for a publisher, but if you set up a site and never opened the bot settings, you may have made a decision without ever making it. Vercel’s firewall has a managed ruleset that can deny known AI crawlers whatever your robots.txt says (Vercel, n.d.).

Blunt blocking can also backfire. Anthropic warns that blocking its IP addresses can stop its bots from even reading your robots.txt, so the choice you meant to express never arrives (Anthropic, n.d.). Perplexity, for its part, recommends permitting requests from its published IP ranges if you want to appear in its results (Perplexity, n.d.).

Next is the road.

Even a bot that gets past the guard still has to walk to the right page. Vercel and MERJ found that ChatGPT’s crawler spent 34.82% of its fetches on pages that returned a 404, with Claude’s close behind at 34.16%, and ChatGPT spent another 14.36% following redirects. Googlebot, in the same data, spent 8.22% and 1.49% (Zecchini et al., 2024). The researchers could not pin down the exact cause, though many misses were old asset paths. Their advice was to keep redirects tidy, sitemaps current and URL patterns consistent (Zecchini et al., 2024). Picture inviting a guest to your house and giving them directions to the wrong compound one time in three.

Finally, the view.

We settled this in the last section, but it belongs on the fence all the same. With an open gate, a friendly guard and a clean road, the crawler still needs your words to be in the HTML it receives. The same study recommends server-rendering your main content, your meta information (titles, descriptions, categories) and your navigation, and leaving client-side rendering for extras like view counters and chat widgets (Zecchini et al., 2024).

Want to know how your own fence looks? Run this against any page:

curl -A "GPTBot" -i https://yoursite.com/your-page

Check the status line first, then search the output for a sentence you wrote. If your words are there, the view is clear. One honest caveat is that this only borrows the bot’s name tag. Google itself notes that user agent strings can be spoofed (Google, n.d.), so a firewall that verifies real IP addresses may treat your test as an impostor. Check the bot analytics in your CDN or host as well.

Only when all four boards stand can we ask the question that actually matters: when a crawler finally hears you, what does it hear?

What Are You Actually Saying?

So, let’s assume the fence can be seen. The gate is open, the guard is friendly, the road is clean and your words are sitting right there in the HTML. A crawler walks in and starts reading. What exactly does it hear?

This is where most of us get embarrassed, because we have never read our own site like a stranger. Remember the tappers and listeners from the naming post? It is the same trap. You know what your product does, so every vague line on your page sounds clear to you. The crawler has no such luck. It only has the words.

So let’s go through them in the order a stranger meets them.

Start with the title. Open your homepage and look at the tab at the top of your browser. Read only that, nothing else on the page. Could someone who has never heard of you tell what you do? “Home” can’t, and “Welcome to our website” can’t either (yes, I am looking at you). Compare those with “Jollof catering for weddings and office events | Ade’s Kitchen”, which tells a person, a search engine and an AI model the same thing in one line.

Then comes the headline and whatever sits under it. A headline can be playful, and ours is: “Is your name actually unclaimed?” That one is for humans, and it works. A crawler can’t do much with a question, though, so the line right under it does the explaining: “Generate names for your idea. Then check domains, social handles, and AI associations before you build around one.” Be as playful as you like up top, as long as a plain sentence follows it.

Next, the first paragraph. Try this. Copy only the first two sentences of a page and paste them somewhere alone. If a stranger can still tell who the page is for and what it gives them, you pass. If it only makes sense with the rest of the page around it, you have some work to do. If you read my post on HyDE vs query rewriting, you know a RAG system lives or dies on finding the right piece of text for a question. So write your opening lines as if they will be read alone, because sometimes they will be.

Finally, the small words, which are the ones we never proofread. Headings should say what is inside the section, so “How we check a name” beats “Our story”. Links should say where they lead, so “Read the pricing breakdown” beats “click here”. Images need alt text that describes what is in them. And each thing should have one name. If your product is “Unclaimed by AI” in one place, “UnclaimedByAI” in another and “Unclaimed” in a third, a crawler cannot be sure it is reading about one product (I know, I have been guilty of this one myself; how I discovered it is a story for another day).

Now read your own page again, tab first, then headline, then the first two sentences. Does it say what you meant to say? If it does, good. If it doesn’t, the crawler has been listening to something else all along. And even a clear message is easier to trust when someone stands behind it.

Who is Doing the Crowing?

We have talked about whether the crawler can reach you, and whether it understands what you said. There is a third thing any listener wants to know before it believes a crow, and it is the same thing you would want to know about a stranger shouting from a fence: who is this, and when did they last say something?

Three things answer that. I call them the name on the door, the date on the wall and the label on the box.

Start with the name on the door. Open one of your own posts and look at where it begins. Is there a name? If you click it, does it go anywhere? Google asks these exact questions in its guidance on helpful content: whether it is obvious who wrote the page, whether there is a byline where you would expect one, and whether that byline leads to background about the author (Google, 2026a). A name that links to a real About page, with real work and a real profile behind it, gives a reader (and a crawler) something to check. The same guidance warns against invented authors, AI-generated headshots and made-up credentials, and says they make a page untrustworthy (Google, 2026a). So the rule is simple: be a person who can be found.

Next, the date on the wall. Put a visible date near the top and label it with words like “Published” or “Last updated”, then make sure your structured data says the same thing (Google, 2025a). Google says it does not lean on one date alone, so it compares several signals and likes them to agree. It also suggests removing other stray dates from the page if the wrong one keeps getting picked (Google, 2025a). And please don’t fake it. Changing a date to look fresh when nothing changed is listed among the warning signs (Google, 2026a). (Yes, I have seen “Updated today” on a post from 2019. We all have.)

Finally, the label on the box, which is structured data. It is a small block of code that sits in your page and tells machines, in a tidy shared vocabulary, what the page is. Here is what one looks like for this very post:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "On-Page Signals That AI Crawlers Look For",
  "datePublished": "2026-10-21",
  "dateModified": "2026-10-21",
  "author": {
    "@type": "Person",
    "name": "Victor Ohachor",
    "sameAs": "https://linkedin.com/in/victorohachor"
  },
  "publisher": {
    "@type": "Organization",
    "name": "Unclaimed by AI",
    "url": "https://www.unclaimedbyai.com"
  }
}
</script>

Read it out loud and it says: this is a blog post with this headline, written by this person (who also lives at this profile), published on this date, by this organization. Google says it uses structured data to understand a page and to learn about the people and companies named in it, and that it can make general use of sameAs (Google, 2025b). Three rules go with it. Only mark up what a visitor can actually see on the page. Give fewer properties that are complete and correct instead of many sloppy ones. And test it with Google’s Rich Results Test before you trust it (Google, 2025b).

Now the part I would rather not skip, even though it would make a more exciting post. I would love to tell you that this block is a megaphone for AI.

Google’s own guidance on its generative AI features says structured data is not required, that there is no special markup to add, and that overfocusing on it is a myth, while still recommending it for rich results (Google, 2026b). As for the other listeners, I could not find anything in OpenAI’s, Anthropic’s or Perplexity’s crawler documentation about reading schema, so I won’t promise you what they do with it. Think of it as good labelling rather than a magic spell.

One important tip, since we met the JavaScript problem earlier. Google says it can read JSON-LD that a script injects after the page loads (Google, 2025b), but most of the other crawlers never run your scripts (Zecchini et al., 2024). So put the block in the HTML your server sends. If you are on Next.js like we are, that just means rendering the script tag in a server component.

Want to check yours? Press Ctrl+U (only works if you are a laptop, desktop, or tablet that has a CTRL key) on any of your pages to see the raw source, then search for ld+json. If you find it, paste the page’s address into the Rich Results Test and see what it says.

So now we have the crow, the one crowing, the date and the labels. Which of them matters most?

Which Crow Wins?

Here is the question you have been waiting for, and I would be lying if I handed you a leaderboard. Remember the scoreboard backlinks used to give us? Nobody has published one for AI, and Google says no third-party tool has access to its internal ranking or AI systems (Google, 2026b). So be careful with anything that promises to show you the inside.

What I can offer is an order of operations, and it follows the fence. Each step answers a different question.

1. Can they reach you? The gate, the guard, the road and the view. A crow nobody can hear wins nothing, so this comes first.
2. Do they understand you? Your title, your headline and your first two sentences, read by a stranger.
3. Do they have a reason to trust you? A name on the door, a date on the wall and a label on the box.

But there is a fourth crow, and it beats the other three. Google’s guide says that creating content people find unique, compelling and useful will likely shape your presence in AI search more than anything else in the guide (Google, 2026b). It separates commodity content, the kind that could have come from anyone (its example is “7 Tips for First-Time Homebuyers”), from non-commodity content that carries a first-hand experience nobody else has (Google, 2026b).

Think about the naming post. A stranger typed a baby-naming query into a tool built for products, and it sent us back to rebuild our whole pipeline. Nobody else could have written that post, because nobody else’s server saw that stranger. That is the crow I mean. Anyone can stand on a fence and crow, but the one that wins is the one only you could have made.

While we are being honest, Google also says you can skip chunking your content into tiny pieces, writing differently just for AI, hunting for “mentions” across the web, and creating llms.txt files, at least for its own search features (Google, 2026b). That is Google speaking about Google, so other systems may differ. It does mean your time is better spent on the four crows above.

In Closing...

Here is a fifteen-minute checklist to do today:

1. Open your robots.txt and read it as each bot would.
2. Check your host’s or CDN’s bot settings, so you know who is being turned away at the guard.
3. Run curl -A "GPTBot" -i on a page and look for one of your own sentences in the output. If you don't know your way around a terminal, use Curlie.
4. Read your title, headline and first two sentences as a stranger.
5. Check that your posts have a clickable name, a labelled date and a JSON-LD block.
6. Write something only you could write.

To see how it is going, Google’s Search Console now has a report on how your content shows up in its generative AI features (Google, 2026b). Pair it with your CDN or server logs to see which AI bots are actually visiting.

References

Anthropic. (n.d.). Does Anthropic crawl data from the web, and how can site owners block the crawler? Anthropic Help Center. Retrieved October 21, 2026, from https://support.claude.com/en/articles/8896518

Cloudflare gives website owners new power over AI crawlers. (2026, July 26). Northeast Times. https://northeasttimes.com/2026/07/26/cloudflare-gives-website-owners-new-power-over-ai-crawlers/

Cloudflare hits the kill switch on AI crawlers—and an entire industry cheers. (2025, July 1). Decrypt. https://decrypt.co/328057/cloudflare-hits-kill-switch-ai-crawlers-entire-industry-cheers

Google. (n.d.). List of Google’s common crawlers. Google for Developers. Retrieved October 21, 2026, from https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

Google. (2025a, December 10). Influence your byline dates in Google Search. Google Search Central. https://developers.google.com/search/docs/appearance/publication-dates

Google. (2025b, December 10). Introduction to structured data markup in Google Search. Google Search Central. https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data

Google. (2026a, October 5). Creating helpful, reliable, people-first content. Google Search Central. https://developers.google.com/search/docs/fundamentals/creating-helpful-content

Google. (2026b, July 10). Optimizing your website for generative AI features on Google Search. Google Search Central. https://developers.google.com/search/docs/fundamentals/ai-optimization-guide

Koster, M., Illyes, G., Zeller, H., & Sassman, L. (2022). Robots Exclusion Protocol (RFC 9309). Internet Engineering Task Force. https://www.rfc-editor.org/rfc/rfc9309

OpenAI. (n.d.). Overview of OpenAI crawlers. OpenAI Developers. Retrieved October 21, 2026, from https://developers.openai.com/docs/bots

Perplexity. (n.d.). Perplexity crawlers. Perplexity Docs. Retrieved October 21, 2026, from https://docs.perplexity.ai/docs/resources/perplexity-crawlers.md

Vercel. (n.d.). Agentic commerce readiness. Vercel Knowledge Base. Retrieved October 21, 2026, from https://vercel.com/kb/guide/agentic-commerce-readiness

Zecchini, G., Moore, A. A., Ubl, M., Siddle, R., & others. (2024, December 17). The rise of the AI crawler. Vercel. https://vercel.com/blog/the-rise-of-the-ai-crawler