What is llms.txt, and how does your site show up in ChatGPT, Claude and Perplexity?

llms.txt is a proposed file that sits in your site's root directory and summarizes your site for AI systems in Markdown; it is not an official standard, and there is no guarantee that AI companies will read it. What actually determines whether your site appears in ChatGPT, Claude or Perplexity answers is whether these companies' crawlers can reach your pages and read the information on them. That access is governed by your robots.txt file and, if you use one, your CDN's bot settings.

What is llms.txt?

llms.txt is a proposal published by Jeremy Howard at llmstxt.org on September 3, 2024. The page states that the proposal was updated on August 10, 2026 and that this is the second version. The goal is to let language models quickly reach a site's most important information without wading through complex HTML pages.

According to the proposal, the file lives at /llms.txt and is written in Markdown. Its structure, in order, is:

1. A # Title line with the name of the site or project. This is the only required section in the proposal.
2. A blockquote starting with > that gives a short summary of the site.
3. Optional free-form description paragraphs.
4. Lists of links to important pages under ## Section name headings. A short description can be added next to each link.
5. By convention, a section called ## Optional for more secondary information.

The proposal also recommends offering a clean Markdown version of important pages at the same address with .md appended.

Is it supported?

llmstxt.org states that OpenAI, Anthropic and Gemini publish llms.txt files for their own developer documentation. However, a company publishing this file for its own documentation does not mean its crawlers read llms.txt files on other sites. The OpenAI, Anthropic and Perplexity crawler pages we cite below contain no commitment that these companies use llms.txt files on websites.

So treat llms.txt as an easy-to-prepare, harmless addition. Your site's core information should still be readable on your pages themselves.

AI crawlers: training, search and user requests

AI companies do not come to your site with a single bot. According to their official documentation, both OpenAI and Anthropic crawl under different names (user agents) for different purposes. Perplexity also runs its search index and user requests through separate bots.

The practical meaning of this distinction is that you may not want your content used for model training, yet you may want your site to appear as a source when a user asks ChatGPT or Claude about your industry. These two preferences can be managed independently of each other.

• OpenAI writes that OAI-SearchBot is used to show sites in ChatGPT's search features and GPTBot to crawl content that may be used to train its foundation models, and states that each setting is independent of the others. It also notes that because ChatGPT-User is initiated by a user action, robots.txt rules may not apply to it.
• Anthropic states that blocking ClaudeBot is treated as a request to exclude your site's future content from training datasets, while blocking Claude-SearchBot and Claude-User may reduce your site's visibility in search answers. According to Anthropic, all three bots respect robots.txt rules.
• Perplexity writes that PerplexityBot is used to show sites in Perplexity search results and does not collect content to train foundation models. Perplexity-User, however, generally ignores robots.txt rules because it acts on a user's request.

How do you control this with robots.txt?

robots.txt is a plain text file in your site's root directory that tells bots which paths they may crawl. Rules are written by user-agent name. For example, a preference that closes your site to model training but keeps it open to search could be expressed like this:

• User-agent: GPTBot followed by Disallow: /
• User-agent: ClaudeBot followed by Disallow: /
• User-agent: OAI-SearchBot followed by Allow: /
• User-agent: Claude-SearchBot followed by Allow: /

This is not a recommendation, only a syntax example; which bots you allow is a business decision. Keep a few points in mind:

• Anthropic states that robots.txt must be edited separately for each subdomain you want to block. If you have www and blog subdomains, check both.
• Anthropic does not recommend blocking by IP address; because this also prevents the bot from reading your robots.txt file, it may not guarantee that your preference is applied consistently.
• As you saw above, some bots that act on user requests may not follow robots.txt. So think of robots.txt as a statement of preference, not an access lock.

Check your CDN's bot settings

If you serve your site behind a CDN, a bot you allowed in robots.txt may be blocked at the CDN layer. Cloudflare, for example, offers the option to block AI bots by dividing them into three groups based on their behavior:

• Search: crawlers that collect or index your content to answer questions later.
• Agent: automated activity acting in real time on behalf of a person; for example, bots that fetch pages for chat assistants and agents that use a browser.
• Training: crawlers that take your content to train or fine-tune a model; mixed-purpose crawlers used for both training and search are also in this group.

According to Cloudflare's documentation, for each group you can choose to block on all pages, block only on pages that show ads, or not block. If you use a different CDN or firewall, check your provider's documentation for a similar setting. What matters is that your robots.txt preference and your CDN setting do not contradict each other.

Short checklist

1. Keep key information readable in HTML. What you sell, whom you serve, your pricing approach and your contact details should be written in the page text. Text that exists only inside an image or is loaded by JavaScript after the page loads may not be readable by every crawler.
2. Have a clear summary page. Your "About us" page or the first paragraph of your product page should describe your company accurately in two or three sentences. AI assistants are well suited to quoting clear sentences like these.
3. Read your robots.txt file. Open yourdomain.com/robots.txt in your browser and check whether there is a line that unintentionally blocks all bots or one of the bots above.
4. Check your CDN and firewall settings. If blocking of AI bots is turned on, make sure it is a deliberate choice.
5. Add llms.txt as an optional extra. A simple file with your site's name, a short summary and links to your most important pages is enough. Make sure it does not contradict your pages and stays up to date.
6. Monitor the results. Track which bots visit your site in your server logs or with a traffic tool. A bot that does not run JavaScript does not show up in visitor counters that run only in the browser, so server-side logs are a more reliable source.