addAI.dev

addAI.dev·Case StudyView as Markdown

By Jay·September 2026·Updated October 2026

How to Make Your Website Visible to AI

#ClientStory#AIVisibility#SmallBusiness

A few weeks ago, Christine, who owns Willow & Co. Hair Studio, asked us a question we're hearing more and more:

“If someone asks ChatGPT to find them a hair salon, would we even show up?”
-- Christine, Willow & Co. Hair Studio

She wasn't being paranoid. People already ask AI assistants for recommendations the way they used to ask a friend or scroll Google Maps. Some of those assistants can now browse the web live, read a site and even fill out a booking form on a client's behalf. If your site is invisible to that traffic, or actively hard for an AI agent to parse, you're losing business you'll never see in your analytics, because it never got the chance to happen.

Willow & Co.'s site looked great to a human. It looked like nothing in particular to an AI agent. Here's what we actually did about it, and how we sequenced the one fix that was worth doing right rather than fast.

Where we started

We ran Willow & Co.'s site through Cloudflare's Agent Readiness scan (the same tool at isitagentready.com that powers the checks below). It scored Level 0, Not Ready. The scan covers a lot of ground (API and agent-login discovery, even commerce protocols), but five of its checks set the basic house rules for AI agents:

  1. robots.txt
  2. a sitemap
  3. AI bot rules
  4. Content Signals
  5. Markdown negotiation

Together they tell an agent who you are and where it's allowed to go. Those five are where a small business site starts, and all of them are fixable at the DNS/hosting level without touching a line of the site's actual design.

Willow & Co.'s site already used Cloudflare's free plan. That comes up again in steps 3 to 5: how Cloudflare identifies AI bots, its AI bot settings and a Markdown feature that needs a paid plan.

1. The Foundation

robots.txt is the foundation: two of the other four fixes (AI bot rules and Content Signals) live inside it, and it's where crawlers look for your sitemap. Per RFC 9309, it lives at /robots.txt as UTF-8 text/plain; serve it with a 200 status so crawlers actually read it:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /booking-api/

Sitemap: https://willowandcohair.com/sitemap.xml
robots.txt

That's the whole requirement: valid User-agent/Allow/Disallow groups, and a Sitemap: line if one exists. Willow & Co.'s internal booking dashboard and API live under /admin/ and /booking-api/ - no reason to invite any crawler in there. (This file only asks crawlers to stay out; keeping /admin/ private is the login's job.)

NOTE: This baseline blocks /booking-api/ for every crawler, including any AI agent that could otherwise complete a booking live for a visitor. That's deliberate: with no bots named yet, there's no way to tell a helpful one apart from any other, so the safe default is closed to all of them. Once bots are named individually below, we open /booking-api/ back up specifically for the ones built to act live on a visitor's behalf.

2. The Map

The sitemap is just as simple: a plain XML file per the Sitemaps protocol, listing the canonical URL for every public page:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://willowandcohair.com/</loc></url>
  <url><loc>https://willowandcohair.com/services</loc></url>
  <url><loc>https://willowandcohair.com/book</loc></url>
  <url><loc>https://willowandcohair.com/contact</loc></url>
</urlset>
sitemap.xml

The easy trap here isn't setup, it's upkeep. A sitemap that still lists a page you took down six months ago is arguably worse than having no sitemap at all - it actively misleads whatever's reading it. Whatever you use to publish new pages needs to touch this file too, or it decays quietly.

3. Welcoming the right AI agents

A wildcard Allow: / already covers every crawler, AI ones included. So why bother naming them? Two real reasons:

The catch: bot names go stale fast. We initially reached for "Claude-Web" from an older reference, but the current, verified identifier is ClaudeBot (Anthropic also runs Claude-User and Claude-SearchBot for different purposes). Here's what we actually used, checked against Cloudflare's current bot reference:

User-agent: GPTBot
Allow: /
Disallow: /admin/
Disallow: /booking-api/

User-agent: ClaudeBot
Allow: /
Disallow: /admin/
Disallow: /booking-api/

User-agent: Google-Extended
Allow: /
Disallow: /admin/
Disallow: /booking-api/

User-agent: PerplexityBot
Allow: /
Disallow: /admin/
Disallow: /booking-api/

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /booking-api/
robots.txt (named groups, plus the wildcard group from step 1)

One important detail: per RFC 9309, a bot that finds its own name follows only the groups that name it. The * group is just a fallback for bots nobody named. So a named bot does not inherit the wildcard's restrictions: repeat the same Disallow lines in every named group, or you've quietly opened /admin/ back up to each bot you named.

A wildcard alone is actually more future-proof against brand-new bots that show up next year, since named rules only cover what you remembered to list. Do both.

There's a second reason naming bots matters, and it's the one that almost bit us: several of these vendors run two different bots under related names - one for training or search indexing, and a separate one that fetches a page live, in real time, on behalf of one specific person mid-conversation.

Blanket-disallowing /booking-api/ across every group above, without carving out the -User bots, blocks the exact scenario this post opened with (an assistant completing a booking live, for a real visitor, on request) for every live agent that honors the file.

The fix: allow the live-agent bots through to the booking API specifically, each kept as its own group (each repeating Disallow: /admin/, since named groups don't inherit the wildcard's rules) rather than folded into the training-bot groups:

User-agent: ChatGPT-User
Allow: /
Allow: /booking-api/
Disallow: /admin/

User-agent: Claude-User
Allow: /
Allow: /booking-api/
Disallow: /admin/

User-agent: Perplexity-User
Allow: /
Allow: /booking-api/
Disallow: /admin/
robots.txt (continued)

Before making that trade, it's fair to ask: does keeping /booking-api/ closed to GPTBot and OAI-SearchBot quietly stop the live -User agents from ever finding the endpoint in the first place? No - and this is exactly the kind of assumption that's easy to get backwards. OpenAI's own developer docs state plainly that ChatGPT-User "is not used for crawling the web in an automatic fashion" and instead fetches a page in real time, at the moment a user asks a question - it doesn't depend on anything GPTBot or OAI-SearchBot crawled beforehand (OpenAI: Overview of OpenAI Crawlers). Anthropic's documentation describes the same three-way split, with Claude-User as its live fetcher (Anthropic: Does Anthropic crawl data from the web?).

A live agent finds the booking endpoint the same way a human visitor would.

The endpoint is sitting right there in the HTML of the booking page the agent was already asked to open, not in a search index built by the training bots. Keep that page itself crawlable, and disallowing /booking-api/ for the indexing bots costs nothing.

One more vendor detail, on whether these live fetchers read robots.txt at all:

Either way, the file is the one place Willow & Co.'s intent is written down for all three live agents, and it's the rule Anthropic's live agent actually follows.

If you're on Cloudflare: AI Crawl Control's Security tab gives you the same outcome as a dashboard toggle per bot. On the free plan it recognizes crawlers by their User-Agent string (which anyone can fake); upgrading adds Cloudflare's Bot Management detection to check who's really asking. That matters a lot more if your policy is to block certain bots; it matters less if, like Willow & Co., your policy is to let everyone in.

4. Content Signals: stating your terms

A newer, still-informal convention for declaring how your content may be used, separate from whether it may be crawled at all. It's one more line inside a group, and we put it in the wildcard group, the same place Cloudflare's own managed file puts it:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /booking-api/
Content-Signal: search=yes, ai-input=yes, ai-train=yes
robots.txt (the wildcard group, with its new line)

Three categories:

Willow & Co. said yes to all three - it's public marketing copy, and being cited in a live AI answer ("try Willow & Co., they're highly rated for highlights") is exactly the visibility this whole exercise is for.

Google's John Mueller has said this directive currently has no effect on any crawler or LLM he's aware of, calling it something "made up by a CDN." It's a Cloudflare-originated proposal, not yet adopted by anyone enforcing it. Cheap to add, correctly implemented, currently symbolic. We added it anyway, because it costs nothing and the standard may still catch on.

If you're on Cloudflare, its settings can rewrite this file for you. Cloudflare sorts AI bots into three policies (Search, Agent and Training), set under Security Settings, a different screen from the per-bot toggles in step 3. Search and Agent can each be Allow, Block on pages with ads or Block. Training adds a fourth option, Disallow AI Training, which blocks training crawlers at Cloudflare's edge but still lets crawlers that also power search (Googlebot, Bingbot, Applebot) in for search. A toggle called Bot Preference Sync writes any Block or Disallow choice into your robots.txt as a managed section at the top of the file you wrote. Leave all three on Allow and it adds no blocking rules.

Check your settings rather than assume them: since September 15, 2026, new Cloudflare domains whose owners say they show ads start with Training set to Disallow AI Training and Agent set to Block on pages with ads.

Here's what that Training default would do to Willow & Co.'s file, going by Cloudflare's documented example:

A robots.txt rule is a request, not a lock.

The result is a file that says one thing and a firewall that does another. The firewall wins: RFC 9309 says robots.txt rules "are not a form of access authorization," and Cloudflare's documentation calls compliance voluntary. The edge block is the part that actually stops a crawler.

Since Willow & Co. said yes to search, AI input and training, the right choice is the simplest one: keep all three Cloudflare policies on Allow. Training's plain Block setting is the one to avoid for any site that wants search traffic: it also blocks Googlebot, Bingbot and Applebot, which cuts a site out of ordinary search as well.

After any change to Cloudflare's AI settings, re-check the live file: open yoursite.com/robots.txt in a browser and look near the top (after Cloudflare's Content Signals Policy comments) for a block starting # BEGIN Cloudflare that you didn't write. Our AI Visibility Scan reads that same live file (Cloudflare's additions included) and checks whether its combined rules still let AI search crawlers reach your key pages. Reconciling everything beyond that (training-bot rules, Content Signal values and Cloudflare's edge settings) is the hands-on part of a client engagement.

5. Markdown for Agents: saved for last, on purpose

This is the one still ahead of us for Willow & Co., and it's arguably the highest-value fix on the list: it's what actually lets an AI agent read clean, structured pricing and booking details instead of scraping them out of dense HTML. We're not skipping it. We're sequencing it, which is a different thing.

The ask: when a request comes in with Accept: text/markdown, return a clean Markdown version of the page instead of dense HTML, while browsers keep getting normal HTML by default. Cloudflare has a managed version of this - flip a switch in the AI Crawl Control section of the dashboard, and Cloudflare converts HTML to Markdown at the edge automatically.

Concretely: a browser's ordinary Accept: text/html request still gets the page exactly as it renders today. An agent that sends Accept: text/markdown instead gets a lean, structured stand-in for the same content, served from the same URL:

GET /services HTTP/1.1
Accept: text/markdown
Request
HTTP/1.1 200 OK
Content-Type: text/markdown
Vary: Accept

# Services
- Cut & style - $98
- Highlights - $198
Response

No new page, no separate URL to maintain - one representation of the content, negotiated by a header most human visitors never send. The Vary: Accept line matters more than it looks: without it, an edge cache can serve the Markdown response to the next browser that asks for the same URL, or vice versa.

The judgment call here matters more than the fix itself: that managed switch requires a Pro or Business plan (Pro is $20/month billed annually, $25/month billed monthly), and it bundles in a pile of other features a lot of small sites don't need. The alternative is to build it ourselves on a free hosting tier - genuinely doable, no paid plan required, using standard HTTP content negotiation (RFC 9110), the same mechanism Cloudflare's switch uses. That's the route we're taking for Willow & Co., because it costs Christine nothing extra and we already have the pieces in place.

A Markdown mirror of a page is only useful if it actually matches the page.

The one thing we're waiting on is timing, not budget: we're holding it until Willow & Co.'s new pricing is final. Build it against numbers that are about to change again, and you've built something that quietly lies to every agent that reads it. So we're building it once, correctly, right after the new pricing locks in, instead of twice.

Scanned again on October 14, 2026, the site passes four of those five house-rule checks and sits at Level 2, Bot-Aware. Markdown negotiation is the one still open, and it's the single requirement the scan lists for Level 3, Agent-Readable.

That's the pattern we'd want any small business owner to walk away with: a cheap, verified win shipped immediately, and a bigger win built once, at the right moment, instead of rushed twice.

Where do you stand? Can AI find your site?

Run our own AI Visibility Scan: paste your URL and, in seconds, get a free check-by-check breakdown of steps 1 to 4 above, plus a few related checks (including structured data and whether agents can find a Markdown version of your pages). isitagentready.com covers similar ground if you want a second read.

Acting on what these scans recommend means DNS-level config, exact RFC syntax and Cloudflare settings that can quietly rewrite your robots.txt. None of that should require a hair salon, a bakery or a landscaper to hire staff. That's the part we do.

Get in touch