If you run a website, a blog or an online store, a small text file on your server just got a little more powerful. OpenAI announced on Friday, October 30, that it will honor a common signal in robots.txt that lets website operators express their preferences about whether their content is used for foundational generative AI training, and communicate that choice to multiple AI companies at once.
Here is what was announced, what it means in practice, and what it does not change, based on OpenAI’s announcement and the work of the IETF AI Preferences working group.
First, what is robots.txt?
Robots.txt is a plain text file that sits at the root of a website, for example at yoursite.com/robots.txt. For decades, it has been the standard way for site owners to tell automated crawlers, such as search engine bots, which parts of a site they may visit. Following it is voluntary, but major search engines and reputable crawlers respect it.
As AI companies began crawling the web to gather training data, many publishers started using robots.txt to block AI crawlers. The problem: each company uses its own crawler names, so site owners had to track and list every one separately.
What OpenAI announced
OpenAI says it will honor a shared signal in robots.txt for expressing preferences about foundational generative AI training. Instead of writing separate rules for every provider, a site owner can state one preference that participating AI companies agree to follow.
“Previously, website operators would identify individual crawlers and maintain separate rules for each and every provider,” OpenAI wrote. “Having a shared signal across multiple providers reduces that work for operators.”
The company says it aims to implement the signal “in the coming months” and will publish details of its implementation when it does.
Where the signal comes from
The effort builds on the Internet Engineering Task Force’s AI Preferences working group, the standards body work focused on how content owners can express preferences about AI use. OpenAI says it is joining other participants in the working group in announcing the shared signal, and it encouraged other AI providers to honor it too.
What does not change
OpenAI was clear that the new signal will not replace its existing crawler controls. Rules that target its specific crawlers in robots.txt “will continue to apply and be respected,” the company said. Think of the shared signal as an extra, simpler layer rather than a replacement.
What this means for you
- Small site owners get a simpler option. Once the signal is live and documented, one line of preference could reach multiple AI providers.
- You still need to decide. Some creators want their work kept out of AI training, while others see value in being included. The signal is about making that choice easier to express, not making it for you.
- It is still voluntary. Robots.txt has always relied on crawlers choosing to comply. A shared signal only works with providers that commit to it.
- Watch for details. OpenAI has not yet published the exact implementation. Avoid copying unofficial snippets until the official format is documented.
A practical checklist for now
While you wait, it is a good moment to check that your site actually has a robots.txt file, review what it currently allows and blocks, and make sure you are not accidentally blocking search engines you want to be found by. If you use WordPress or another content management system, your SEO plugin may already manage this file for you.
The bigger picture
The relationship between AI companies and the open web has been tense, with publishers worried about their work being used without permission. A shared, standards-based signal will not resolve every dispute, but it is a practical step toward clearer rules that work across the industry.
Wrapping up
For now, the headline is simple: OpenAI has committed to respect a single, common way for websites to say how they feel about AI training. When the details land in the coming months, site owners will have an easier way to make their choice count across multiple AI providers at once.
Image: BT Italia Settimo Milanese by Davide Alotto via Wikimedia Commons, CC0 1.0
