Paper | Jiménez, Arkko: AI, Robots.txt
slides-aicontrolws-ai-robotstxt-00

Large Language Models (LLMs) and their use of Internet-sourced material present numerous technical, commercial, legal, societal, and ethical challenges. An emerging practice proposes extending the robots.txt file to enable website owners to declare if they wish to "opt-out" from having their site’s content used in training AI models.

This paper explores the topic. We argue that the problem is much broader than the simple opt-out mechanism, given the coming new applications, the many different ways to access training material, different AI techniques, and the need to both facilitate access to training material and enable opting out from it.