Project
Nonprofit organization that crawls the public web and publishes free, open datasets of web pages, widely used to train AI language models.
Anthropic asked a San Francisco court to make OpenAI answer 25 subpoena requests, among them its records on deleted Reddit posts. The motion is set for a November 12 hearing on whether OpenAI must produce its deletion logs and its records on removing posts from trained models.
Reality
OpenAI's ChatGPT chief wrote that its products "are largely substitutive, period," in internal documents made public September 17 in the Times lawsuit. That helps publishers argue over AI search, where Fast Company says the money is, but the labs still set the price.
Reality
Pennyforge's 1 October scan of 185 European sites found 35 AI-crawler opt-outs in robots.txt, every one sitewide. That gives the EU consultation on opt-out signals, closing 3 November, a baseline where three in four sites show crawlers no AI rule at all.
Reality
New York Times lawyers quote a 2023 Microsoft memo calling AI scraping 'the largest theft of labor in human history' in briefs a court unsealed September 17. For teams training on unlicensed data, the case puts dataset custody and internal email in the same record.
Reality
A detector run over 490,000 Common Crawl pages puts a number on synthetic text in the crawlable web, and the number is uneven enough to change how you build a training set.
Reality
A newly unredacted brief in the New York Times' case quotes a Microsoft director calling AI scraping the largest theft of labor in human history. The underlying exhibits are still sealed, so the quotes arrive without their original context.
Perspective Coverage
6 publishersReality
A site logged twenty requests for its unreachable /.env file from a client naming itself ClaudeBot. GreyNoise counted 824 addresses scanning under forged AI crawler names, and saw no request for robots.txt in the whole campaign.
Reality
Before starting a column about getting cited by LLMs, a developer curled medium.com/robots.txt and found GPTBot, ClaudeBot and six other agents disallowed. Common Crawl and Google-Extended are still allowed.
Reality
One site logged 13,491 crawler requests in thirty days and could prove the origin of roughly 2,200 of them, because half the traffic came from vendors that publish no addresses to check against.
Reality
A dev.to census of 1,551,977 job postings across 12 open ATS endpoints found BambooHR's isRemote flag empty in all 50,616 it saw. Having a field and having data in it are separate claims.
Reality