By JaySeptember 2026
The Header We Bet On Just Got Extended
Our last post made a specific bet: our robots.txt ships Cloudflare's Content-Signal field even though, per Google's own John Mueller, it currently does nothing for any crawler. The reasoning was that if Cloudflare keeps building on the field, we'd already be positioned for it rather than retrofitting later. This week gave us a real, checkable data point on that bet, and a second, unrelated change worth not confusing it with.
The field we bet on, extended
Cloudflare's own announcement extended Content-Signal with a new use parameter, letting a site express not just whether crawling is welcome, but what a crawler may do with the content afterward:
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /Three levels: immediate (interact, but store and reuse nothing), reference (index, excerpt, and link back), full (summarize and reproduce). It's the same field, same file, same voluntary mechanism we already ship, just with more to say. That's real evidence the field is actively maintained, not abandoned. It's a smaller bet payoff than it might first look, though: Cloudflare published that extension on July 1, 2026, months before we wrote about the field at all. It didn't happen in response to our post, we just hadn't caught up to it yet.
What actually landed this week
What's genuinely dated to this week is different: September 15, 2026 is when the default settings from that same July announcement activated for new domains onboarding to Cloudflare, Search allowed, Training and Agent blocked by default on ad-monetized pages. Cloudflare also sent existing accounts, ours included, a direct notice that any site still running the old "Block AI Bots" or "Managed Robots.txt" toggle was being moved onto the new three-way settings.
That notice changed nothing on our end, because we never enabled either legacy toggle in the first place, a call we'd already made and documented before this rollout existed. Our hand-rolled robots.txt rules were never in Cloudflare's migration path to begin with.
Two mechanisms, not one
It's easy to read "Cloudflare split AI bot handling into three real categories" and "Cloudflare extended the preference field" as the same story. They aren't, and mixing them up is the one thing that would make this post worth less than it should be.
Search, Agent, and Training are things Cloudflare enforces at its own network edge. Content-Signal, use= included, is still just a preference.
Search, Agent, and Training are bot-management categories Cloudflare itself blocks or allows at the network edge, real enforcement, no crawler cooperation required. Content-Signal, including the new use= value, is still exactly what Cloudflare's own writeup calls it: a preference, not a block. Compliance runs on Verified Bots choosing to honor it, tracked through Cloudflare's BotBase reputation system, not on anything technical stopping a bot that ignores it.
Where that leaves the bet
What changed is that the field itself keeps getting more capable, built by the same company running a meaningful share of the web's edge traffic, and our site was already speaking it before this month's rollout made it relevant enough to check. That's the actual shape of the bet: not that the header works yet, but that it's cheap enough to ship early and mature enough, now, to be worth having shipped.
That's the same standard we hold every claim on this site to: check what actually happened before writing about it, not what a headline implies happened.