Researchers with the Data Provenance Initiative examined 14,000 domains and found “a proliferation of AI-specific clauses to limit use.” Between April 2023 and 2024, restrictions were added on five per cent of the data in the widely used C4, RefinedWeb and Dolma datasets and 25 per cent from critical sources, the study says. (The Logic)

Talking point: Large language and other foundational models train on huge quantities of content, enabling them to produce responses to queries, code and other outputs. But Data Provenance researchers found that website publishers are increasingly adding language to their robots.txt—files that tell web crawlers what they can and can’t take—specifically telling AI developers they’re not welcome to the data. ChatGPT-maker OpenAI was the most restricted of the five commercial organizations assessed, at 25.9 per cent, with Toronto’s Cohere lower at 4.9 per cent. The study calls it an “emerging crisis in data consent, foreclosing much of the open web” for both for-profit and academic AI use.