LLMS.txt
In case you are interested in it, there is a proposal going around for a llms.txt text file for websites. Basically like robots.txt but for context for Large Language Models. Thought it would be worth sharing so more people will adopt it.
Here is the Proposal for the standard:
https://llmstxt.org/
A newer proposal (2024).
Targeted at Large Language Models (LLMs).
Meant to provide structured guidance for training, citations, attribution, etc.
Supported by: some in the LLM training/data ethics community (not as widespread yet, but gaining traction).
Here’s mine if you are curios and want to see an example:
https://toasterbotnet.com/llms.txt
AI.txt
There also another thing called ai.txt
Proposed around 2023.
General-purpose:
tells any AI crawler how you want your site’s data to be used (or not used).Think of it as robots.txt, but for AI crawlers.
Supported by: OpenAI, Anthropic, Perplexity, etc.
That is a bit more like the old robots.txt
Here is my example:
https://toasterbotnet.com/ai.txt