addAI.dev

addAI.dev·Case Study

By Jay · Last verified September 2026

How to Make Your Website Visible to AI

#ClientStory#AIVisibility#SmallBusiness

A few weeks ago, Christine, who owns Willow & Co. Hair Studio, asked us a question we're hearing more and more:

“If someone asks ChatGPT to find them a hair salon, would we even show up?”

She wasn't being paranoid. People already ask AI assistants for recommendations the way they used to ask a friend or scroll Google Maps. Some of those assistants can now browse the web live, read a site, and even fill out a booking form on a client's behalf. If your site is invisible to that traffic, or actively hard for an AI agent to parse, you're losing business you'll never see in your analytics, because it never got the chance to happen.

Willow & Co.'s site looked great to a human. It looked like nothing in particular to an AI agent. Here's what we actually did about it, and how we sequenced the one fix that was worth doing right rather than fast.

Where we started

We ran Willow & Co.'s site through Cloudflare's Agent Readiness scan (the same tool at isitagentready.com that powers the checks below). It landed on the scan's Quick Wins checklist at Level 1, 0 of 5 complete - the five items Cloudflare flags as the baseline for setting basic house rules for AI agents, so they know exactly who you are and where they're allowed to go: robots.txt, a sitemap, AI Crawler Rules, Content Signals, and Markdown Negotiation. All five are fixable at the DNS/hosting level without touching a line of the site's actual design.

1. The Foundation

Every one of the other four fixes builds on this file existing and being valid - Cloudflare flags it, correctly, as High impact. Per RFC 9309, it has to be served at /robots.txt, as text/plain, with a 200 status:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /booking-api/

Sitemap: https://willowandcohair.com/sitemap.xml
robots.txt

That's the whole requirement: valid User-agent/Allow/Disallow groups, and a Sitemap: line if one exists. Willow & Co.'s internal booking dashboard and API live under /admin/ and /booking-api/ - no reason to invite any crawler in there.

NOTE: This baseline blocks /booking-api/ for every crawler, including any AI agent that could otherwise complete a booking live for a visitor. That's deliberate: with no bots named yet, there's no way to tell a helpful one apart from any other, so the safe default is closed to all of them. Once bots are named individually below, we open /booking-api/ back up specifically for the ones built to act live on a visitor's behalf.

2. The Map

Also flagged High impact, and just as simple: a plain XML file per the Sitemaps protocol, listing the canonical URL for every public page:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://willowandcohair.com/</loc></url>
  <url><loc>https://willowandcohair.com/services</loc></url>
  <url><loc>https://willowandcohair.com/book</loc></url>
  <url><loc>https://willowandcohair.com/contact</loc></url>
</urlset>
sitemap.xml

The easy trap here isn't setup, it's upkeep. A sitemap that still lists a page you took down six months ago is arguably worse than having no sitemap at all - it actively misleads whatever's reading it. Whatever you use to publish new pages needs to touch this file too, or it decays quietly.

3. Welcoming the right AI agents

A wildcard Allow: / already covers every crawler, AI ones included. So why bother naming them? Two real reasons:

The catch: bot names go stale fast. We initially reached for "Claude-Web" from an older reference, but the current, verified identifier is ClaudeBot (Anthropic also runs Claude-User and Claude-SearchBot for different purposes). Here's what we actually used, checked against Cloudflare's current bot reference:

User-agent: GPTBot
Allow: /
Disallow: /admin/
Disallow: /booking-api/

User-agent: ClaudeBot
Allow: /
Disallow: /admin/
Disallow: /booking-api/

User-agent: Google-Extended
Allow: /
Disallow: /admin/
Disallow: /booking-api/

User-agent: PerplexityBot
Allow: /
Disallow: /admin/
Disallow: /booking-api/

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /booking-api/
robots.txt (continued)

One important detail: per RFC 9309, a crawler matches one group, the most specific one for its name, not a blend of its own group and the wildcard. So every named group needs to repeat the same Disallow lines as the wildcard group - otherwise you've accidentally opened /admin/ back up to any bot you named explicitly.

A wildcard alone is actually more future-proof against brand-new bots that show up next year, since named rules only cover what you remembered to list. Do both.

There's a second reason naming bots matters, and it's the one that almost bit us: several of these vendors run two different bots under related names - one for training or search indexing, and a separate one that fetches a page live, in real time, on behalf of one specific person mid-conversation.

Blanket-disallowing /booking-api/ across every group above, without carving out the -User bots, blocks the exact scenario this post opened with: an assistant completing a booking live, for a real visitor, on request.

The fix: allow the live-agent bots through to the booking API specifically, each kept as its own group rather than folded into the ones above:

User-agent: ChatGPT-User
Allow: /
Allow: /booking-api/

User-agent: Claude-User
Allow: /
Allow: /booking-api/

User-agent: Perplexity-User
Allow: /
Allow: /booking-api/
robots.txt (continued)

Before making that trade, it's fair to ask: does keeping /booking-api/ closed to GPTBot and OAI-SearchBot quietly stop the live -User agents from ever finding the endpoint in the first place? No - and this is exactly the kind of assumption that's easy to get backwards. OpenAI's own developer docs state plainly that ChatGPT-User "is not used for crawling the web in an automatic fashion" and instead fetches a page in real time, at the moment a user asks a question - it doesn't depend on anything GPTBot or OAI-SearchBot crawled beforehand (OpenAI: Overview of OpenAI Crawlers). Anthropic's documentation describes the same three-way split for Claude-User (Anthropic: Does Anthropic crawl data from the web?). In practice, a live agent finds the booking endpoint the same way a human visitor would - it's sitting right there in the HTML of the booking page it's already been asked to open, not in a search index built by the training bots. Keep that page itself crawlable, and disallowing /booking-api/ for the indexing bots costs nothing.

If you're on Cloudflare: AI Crawl Control's Security tab gives you the same outcome as a dashboard toggle per bot, with one real advantage - it verifies bot identity via request fingerprinting at the edge, rather than trusting a self-reported User-Agent string anyone could fake. That matters a lot more if your policy is to block certain bots; it matters less if, like Willow & Co., your policy is to let everyone in.

4. Content Signals: stating your terms

A newer, still-informal convention for declaring how your content may be used, separate from whether it may be crawled at all:

Content-Signal: search=yes, ai-input=yes, ai-train=yes
robots.txt (continued)

Three categories:

Willow & Co. said yes to all three - it's public marketing copy, and being cited in a live AI answer ("try Willow & Co., they're highly rated for highlights") is exactly the visibility this whole exercise is for.

In the interest of not overselling a fix: Google's John Mueller has said this directive currently has no effect on any crawler or LLM he's aware of, calling it something "made up by a CDN." It's a Cloudflare-originated proposal, not yet adopted by anyone enforcing it. Cheap to add, correctly implemented, currently symbolic. We added it anyway, because it costs nothing and the standard may still catch on.

A real gotcha if you're on Cloudflare: there's a "Cloudflare-managed robots.txt" toggle that adds Content Signals for you automatically. It sounds like a shortcut, but read the fine print - it doesn't replace your file, it prepends its own managed section on top of whatever you already have. If you've already hand-tuned bot rules like we did in step 3, and Cloudflare's managed defaults disagree with yours for the same bot, RFC 9309's group-combining rules mean you can end up with contradictory directives for that bot in one file, with no reliable way to predict which one wins, and no preview before you turn it on. If your own file already passes the check (ours did), there's nothing to gain and a real risk of muddying a policy you set on purpose.

5. Markdown for Agents: saved for last, on purpose

This is the one still ahead of us for Willow & Co., and it's arguably the highest-value fix on the list: it's what actually lets an AI agent read clean, structured pricing and booking details instead of scraping them out of dense HTML. We're not skipping it. We're sequencing it, which is a different thing.

The ask: when a request comes in with Accept: text/markdown, return a clean Markdown version of the page instead of dense HTML, while browsers keep getting normal HTML by default. Cloudflare has a managed version of this - flip a switch in AI Crawl Control's Overview tab, and Cloudflare converts HTML to Markdown at the edge automatically.

Concretely: a browser's ordinary Accept: text/html request still gets the page exactly as it renders today. An agent that sends Accept: text/markdown instead gets a lean, structured stand-in for the same content, served from the same URL:

GET /services HTTP/1.1
Accept: text/markdown
Request
HTTP/1.1 200 OK
Content-Type: text/markdown
Vary: Accept

# Services
- Cut & style - $98
- Highlights - $198
Response

No new page, no separate URL to maintain - one representation of the content, negotiated by a header most human visitors never send. The Vary: Accept line matters more than it looks: without it, an edge cache can serve the Markdown response to the next browser that asks for the same URL, or vice versa.

The judgment call here matters more than the fix itself: that managed switch requires Cloudflare's Pro plan ($20/month billed annually, $25/month billed monthly), and it bundles in a pile of other features a lot of small sites don't need. The alternative is to build it ourselves on a free hosting tier - genuinely doable, no paid plan required, using the exact same content-negotiation approach the standard describes. That's the route we're taking for Willow & Co., because it costs her nothing extra and we already have the pieces in place.

The one thing we're waiting on is timing, not budget. Their pricing is being updated right now, and a Markdown mirror of a page is only useful if it actually matches the page. Build it against numbers that are about to change again, and you've built something that quietly lies to every agent that reads it. So we're building it once, correctly, right after the new pricing locks in, instead of twice.

Re-scan the site today and the same Quick Wins checklist reads Level 1: 4 of 5 complete - every item done except Markdown Negotiation, held back for the reason above, not for lack of trying.

That's the pattern we'd want any small business owner to walk away with: a cheap, verified win shipped immediately, and a bigger win built once, at the right moment, instead of rushed twice.

Where do you stand? Can AI find your site?

Head to isitagentready.com, paste your URL, and run the free scan - two minutes, nothing more technical than copying a link. You'll get a report broken down check by check, the same one we ran for Willow & Co.

Most of what it recommends - DNS-level config, exact RFC syntax, Cloudflare tooling that can quietly conflict with itself - is exactly what a hair salon, a bakery, or a landscaper shouldn't need to hire staff for. That's the part we do.

Get in touch