Skip to main content
External Link
CNN, Reuters, Australia’s ABC, and other news organizations block OpenAI’s web crawler.

The New York Times was among the first to block GPTBot from accessing its data to improve AI models. Any website owner can disallow GPTBot from scraping their website through the robots.txt file. The Guardian reported that other news organizations have also blocked Common Crawl’s CCBot.