Meta's commercial terms, effective October 30, appear to let it scrape sites tied to business accounts for AI training, according to Nieman Lab. With no opt-out, a publisher negotiating payment for its archive is bargaining with a counterparty that may already hold a contractual route to the same text.
Perspective Coverage
5 publishers
- Builder
- Builder 17%
- Operator
- Operator 51%
- Investor
- Investor 32%
Reality
- Evidence58
- Adoption
- Not scored
- Hype gap+30
- Incentives50
- Confidence62
buildOne report1 publisher Meta's October 30 Commercial Terms ask businesses on Facebook to let it crawl their sites for AI training
Meta's Commercial Terms take effect October 30 and ask Facebook business users to permit crawling of their websites for AI training. Meta says robots.txt controls stay as they are, but the permission sits in a contract.
Reality
- Evidence55
- Adoption
- Not scored
- Hype gap+10
- Incentives
- Not scored
- Confidence50
buildOne report1 publisher 69 of 109 top sites that block GPTBot also block ChatGPT's search or browsing agent
One author's October 12 crawl of 740 top Tranco sites found that 69 of the 109 blocking GPTBot also block ChatGPT's search or browsing agent. Those sites opted out of ChatGPT answers as well as training, and the crawl cannot say how many meant to.
Reality
- Evidence55
- Adoption15
- Hype gap0
- Incentives30
- Confidence60
buildOne report1 publisher Cloudflare's new-domain defaults block AI agents on pages that carry ads
Cloudflare blocks AI agent and training traffic by default on ad-carrying pages of domains added since September 15, 2026. Search crawlers stay allowed, so a browsing assistant can be refused a page that a search indexer reads freely.
Reality
- Evidence45
- Adoption
- Not scored
- Hype gap+10
- Incentives
- Not scored
- Confidence40
buildOne report1 publisher A trailing slash in robots.txt would have left pub-trivia.app's dashboard crawlable
Pub-trivia.app's developer warns that Disallow: /dashboard/ would block every dashboard page except the bare /dashboard its homepage footer links to. Blocking a fetch never stops indexing, so the noindex tag has to go live before the Disallow.
Reality
- Evidence45
- Adoption
- Not scored
- Hype gap+5
- Incentives
- Not scored
- Confidence55
Anthropic asks Australia to let AI train on copyrighted work unless owners opt out
Anthropic told an Australian inquiry a broad copyright exemption is off the table and proposed opt-out 'conditional approval' for AI training instead. The ABC and SBS want the onus on AI firms, so the inquiry is now deciding who pays to police training bots.
Reality
- Evidence58
- Adoption
- Not scored
- Hype gap+15
- Incentives75
- Confidence60
buildOne report1 publisher FoxyInvoice's Angular site gave crawlers one word until a build-time prerender filled it in
FoxyInvoice prerenders every public Angular route to static HTML at build time so crawlers that skip JavaScript get the full page text. A build step and a one-line drift check take the place of the always-on SSR server its team declined to run.
Reality
- Evidence40
- Adoption15
- Hype gap+20
- Incentives70
- Confidence40
buildOne report1 publisher Cloudflare's new AI training opt-out is backed by network-level enforcement Apple, Google and Microsoft have agreed to honor
Fewer than 1% of Cloudflare sites block search crawlers while 17% block training. A single mixed-use crawler used to collapse those two into one decision, and the new setting splits it for three named operators.
Reality
- Evidence55
- Adoption35
- Hype gap+20
- Incentives70
- Confidence60
Rendering price in JavaScript keeps a product out of ChatGPT's shopping answers
An ecommerce audit consultant writing in Entrepreneur says most AI crawlers read only a page's initial HTML, so a product page that ranks in Google can return nothing to an assistant fielding around 50 million shopping queries a day.
Reality
- Evidence24
- Adoption33
- Hype gap+38
- Incentives72
- Confidence45
Amazon shuts OpenAI's shopping agent out of its own product pages
Amazon's robots.txt now disallows ChatGPT-User and OAI-SearchBot, and in ZDNET's tests the shopping agent sent a buyer to Walmart, eBay and a retailer called Soaplicity instead of Amazon listings.
Reality
- Evidence64
- Adoption66
- Hype gap+24
- Incentives72
- Confidence58
Amazon blocks the two OpenAI crawlers behind 50 million daily shopping queries
New disallow rules for ChatGPT-User and OAI-SearchBot went into Amazon's robots.txt without an announcement. Amazon earns roughly $56 billion a year from advertising that bills only while shoppers are on its own pages.
Reality
- Evidence68
- Adoption58
- Hype gap+22
- Incentives74
- Confidence66
buildOne report1 publisher A log audit can only catch the nine AI agents that send an HTTP request
Google-Extended and Applebot-Extended are robots.txt tokens that never issue a request, so the nine agents OpenAI, Anthropic and Perplexity document are the only AI traffic a WordPress site can confirm for itself.
Reality
- Evidence74
- Adoption48
- Hype gap−8
- Incentives68
- Confidence60
buildOne report1 publisher Eight names in Medium's robots.txt close training ingestion while retrieval stays open
Before starting a column about getting cited by LLMs, a developer curled medium.com/robots.txt and found GPTBot, ClaudeBot and six other agents disallowed. Common Crawl and Google-Extended are still allowed.
Reality
- Evidence62
- Adoption40
- Hype gap−8
- Incentives68
- Confidence55
buildOne report1 publisher Cloudflare splits search from training by publishing a no-training line in your robots.txt
Cloudflare's Disallow AI Training control lets Googlebot and Applebot keep indexing a site while a directive Cloudflare writes into robots.txt tells them not to train on it. Bingbot is expected to honor it in early 2027.
Reality
- Evidence30
- Adoption25
- Hype gap+25
- Incentives70
- Confidence35
buildOne report1 publisher Three hand-written route lists put /about in the sitemap and behind a login gate
Eight hand-kept lists sat behind every new page on pub-trivia.app. One typed array now feeds the sitemap, breadcrumbs and footer, with unit tests that fail the commit when a description runs long.
Reality
- Evidence58
- Adoption14
- Hype gap+12
- Incentives40
- Confidence55
buildOne report1 publisher The EU AI Act pushes training-data provenance down into the crawler
The AI Act's general-purpose AI obligations became enforceable in August, and the parts of them that touch a data pipeline depend on records only the collector can write while it is fetching. Penalties scale with turnover.
Reality
- Evidence32
- Adoption
- Not scored
- Hype gap+22
- Incentives70
- Confidence44
buildOne report1 publisher A used Raspberry Pi case asked 31% above new while the retail anchor returned no results
With no completed sales, nine live asks running from £35 to £240 and no route to the £26.99 shop price, a self-hosted pricing tool proposed £35.39 and recorded the missing check only in a log line.
Reality
- Evidence50
- Adoption12
- Hype gap+6
- Incentives30
- Confidence55
buildOne report1 publisher A page that ranks can hand GPTBot an empty HTML shell
A dev.to post sets out two ways assistant retrieval fails while ordinary search keeps working, one in robots.txt rules written per user-agent and one in copy that appears only after JavaScript runs. It offers no measurement.
Reality
- Evidence30
- Adoption
- Not scored
- Hype gap+15
- Incentives
- Not scored
- Confidence40
buildOne report1 publisher Pandex hooked a Fortune 500 agent four minutes after claiming a package name from llms.txt
The file exists so an agent can skip your documentation and read the install steps instead. Pandex checked 8,565 of them, found 237 references pointing at names anyone could register, and registered one.
Reality
- Evidence38
- Adoption52
- Hype gap+18
- Incentives62
- Confidence44
buildOne report1 publisher Sorting agent tasks by cost-to-undo decides how much work the agent does alone
One developer's cache-header diff read perfectly and still shipped a dead year-long cache, which is the argument for granting agents autonomy only where git revert plus a redeploy is the whole undo path.
Reality
- Evidence52
- Adoption12
- Hype gap+8
- Incentives22
- Confidence50