Get Started

Say it once: introducing Bot Preference Sync

LaunchSecurityAutomation, Bots & Scraping

First reported · 2mo ago · 1 source

Crest summary — written by AI from the 1 source below; every figure is checked against them · model deepseek-chat

Summary

Cloudflare introduced Bot Preference Sync, a feature that automatically aligns a site's robots.txt file with its AI bot policies for Search, Agent, and Training, removing the need to maintain static files.

Key facts

  • Cloudflare announced Bot Preference Sync on 2026-08-21.
  • The feature aligns robots.txt with AI bot policies for Search, Agent, and Training.
  • It manages which bots access content without maintaining static files.

Why it matters

Site operators can set AI bot access rules in one place instead of editing robots.txt by hand, which reduces the chance of stale or mismatched crawler policies.

Companies

Technologies

  • Bot Preference Syncproduct
  • robots.txtstandard

Original sources (1)

The report this event was built from. Crest of Tech keeps no article bodies: the full text is at each source.

  1. Cloudflare BlogOfficial · Primary
    Say it once: introducing Bot Preference Sync · By Jin-Hee Lee
    Read at Cloudflare Blog
Say it once: introducing Bot Preference Sync — Crest of Tech