What llms.txt Can and Cannot Do for Your Website

Last updated: 24 September 2026.

A useful file can help an agent find an implementation guide instead of navigating dozens of menus. A neglected file can direct the same agent to a retired product, old pricing or a page that no longer exists.

The decision is therefore about publishing and maintaining a useful interface to your content. It is not a shortcut around writing content worth reading.

For a team assessing AI-ready information and data, start with the questions the audience needs answered and the public sources that answer them. Then decide whether an llms.txt entry point would make those sources easier to retrieve.

What is the current proposal?

The llms.txt proposal describes a Markdown overview with a site or project title, optional explanatory material and grouped resource links. Its August 2026 revision is version 2. The proposal remains a convention for consumers that support it, rather than a requirement every AI service must follow. The llms.txt proposal.

Version 2 clarifies path-specific files and discovery links. It supports linking a page to its Markdown alternative and to the llms.txt file describing its section, using standard link relations. It also makes the “Optional” section a convention for secondary material rather than a mechanically enforced instruction to omit it. Version 2 changes.

This distinction matters when you use a generator. A tool may implement an older interpretation, publish only a root file or create a large concatenated export. Inspect what it actually produces and how your intended consumer uses it.

For example, a product's developer documentation and public marketing site may need different entry points. The developer needs authentication, endpoint behavior and error handling. The buyer needs product scope, implementation requirements and current commercial information. Combining everything into one undifferentiated list can make both tasks harder.

Does llms.txt improve Google rankings or AI Overview visibility?

Google's current optimization guide says Google Search does not use llms.txt or similar special files and that maintaining them neither helps nor harms Search visibility or rankings. That is more specific than saying the file is “not required.” Do not sell its presence as a Google ranking improvement. Google's generative AI optimization guidance.

Chrome's Lighthouse documentation does include an llms.txt agentic-browsing audit. However, it treats an absent file returning 404 as not applicable because the file is optional; it flags server errors when attempting retrieval. A browser audit and a Search ranking signal are different things. Lighthouse llms.txt audit.

A site can therefore reasonably maintain a file for agent usability while expecting no ranking effect from the file itself. Assess the value against the actual integration, not a generic promise that “AI engines prefer websites with llms.txt.”

For public search outcomes, use the appropriate platform reports and AI search measurement. A request for the file in server logs does not establish that an answer cited you or that a qualified buyer arrived.

How is it different from robots.txt and a sitemap?

Use each file for its own purpose. Your sitemap helps communicate the public URLs you want search engines to discover. Your llms.txt guide helps an interested consumer navigate selected information. Your robots.txt file communicates crawler access preferences to crawlers that follow it.

Neither an llms.txt exclusion nor a missing link protects private content. An unpublished proposal accidentally deployed at a public URL remains a publication problem even if your directory does not mention it.

Google's robots.txt documentation also cautions that crawl restrictions are not a reliable way to keep a URL out of Search or secure private material. Some blocked URLs can still appear, and security requires actual access controls. Google robots.txt guidance.

Manage crawler purposes separately. OpenAI documents independent controls for OAI-SearchBot, which supports search discovery, and GPTBot, which relates to model training. It separately describes ChatGPT-User as user-initiated retrieval, for which robots.txt rules may not apply. OpenAI crawler documentation.

Anthropic likewise documents distinct training, search and user-requested retrieval agents, with its own stated robots.txt behavior. Check the specific provider's current guidance before copying a universal “allow AI” or “block AI” snippet. Anthropic crawler documentation.

A sensible publishing policy might allow public search discovery while choosing different training preferences. That policy belongs in the relevant crawler configuration and infrastructure, not in a paragraph inside llms.txt.

What should you include for a business website?

Select information that resolves a reader's decision. For a technical services business, that could include service scope, an implementation process, supported systems, a clear pricing explanation and guides answering frequent evaluation questions.

Do not turn the file into a list of every URL simply because the CMS can export them. A category page with no explanatory content, a duplicate campaign landing page and an outdated announcement may add noise rather than useful context.

Write descriptions that distinguish similar pages. “Data guide” does not help a consumer choose between a warehouse architecture tutorial and a tracking implementation checklist. “Explains how to reconcile ad-platform spend with warehouse reporting” does.

An illustrative file for a fictional company might look like this. The example.com addresses are placeholders for actual public Markdown resources you would need to publish:

# Example Growth Systems

> Public guides for evaluating and implementing a marketing data stack.

Commercial details are maintained on the pricing page linked below.

## Planning

- [Implementation scope](https://example.com/docs/scope.md): Deliverables and client responsibilities.
- [Pricing](https://example.com/docs/pricing.md): Current fees and exclusions.

## Implementation

- [Tracking QA](https://example.com/docs/tracking-qa.md): Event checks before activation.
- [Revenue reconciliation](https://example.com/docs/reconciliation.md): Comparing billing and CRM totals.

## Optional

- [Release history](https://example.com/docs/releases.md): Changes to these guides.

The point is the selection and descriptions. Do not publish these placeholder URLs or claim you offer Markdown routes that return a generic HTML fallback. Check each destination before including it.

Avoid copying volatile prices into several summaries. Link to the maintained pricing resource and explain what it covers. If you do include a price in the guide, treat it as another published location that must change when the source changes.

Keep the public page and machine-readable version aligned

A Markdown export should preserve consequential meaning: limitations, dates, eligibility conditions, units and links to evidence. Removing navigation is helpful. Removing the caveat that a feature requires a particular plan can make the export misleading.

Generate both representations from the same content source where practical. Keep a stable identifier for the article and use the publication pipeline to update the HTML page, its Markdown version and the relevant resource list together.

Use genuine content modification dates. Rebuilding a file every day does not mean every linked article was updated that day. A scheduled timestamp refresh can make stale advice look newly verified without changing the underlying facts.

When a page is retired, decide whether to redirect it to a true replacement or remove its entry. A redirect to the homepage usually does not preserve the original purpose. If an old API guide remains available for a supported legacy version, label that version clearly.

A small site may maintain a short file manually. A large content library benefits from a generated index with an editorial selection field. The automation should handle URLs and metadata; a content owner should decide which resources are useful starting points.

Verify the response, not just the file in your repository

Deployment can alter what a consumer receives. A route may return the site's HTML shell with a 200 status, a login screen, an error document or an outdated CDN copy. A file existing in the source tree does not prove the public endpoint works.

Check the public request and its response body. Confirm readable Markdown, the intended status, a suitable text content type and correct encoding. Test the canonical host you actually use, along with any redirect from an alternate host.

Follow the resource links and inspect representative destinations. An HTTP 200 alone is not a content test. Confirm that the response contains the expected title and meaningful article text, including the limitations that affect the answer.

If you publish path-specific files, check that their scope and discovery links match the intended section. For a site under /docs/, do not assume you control the origin root or that every consumer guesses the same location.

Open full diagram

How can you test whether it is useful?

Create a small set of representative questions with known answers and required source pages. For a software company, include a supported feature, a plan restriction, an implementation prerequisite and a question the documentation cannot answer.

Test the same consumer under two defined conditions: its normal entry point and the llms.txt entry point. Record the tool or model version, available retrieval permissions and the documents actually accessed. Keep the questions and evaluation criteria unchanged.

For an illustrative ten-question test, suppose the normal entry point yields six fully supported answers, while the guided entry point yields eight. That is a useful reason to inspect the two improved cases. It is not proof of a universal 33% business improvement, nor a reliable performance estimate from ten questions.

Also examine regressions. The guide might improve API setup answers while directing a pricing question to an old export. A higher aggregate score should not hide a consequential factual error. Fix the source selection or publishing process before expanding the file.

Do not reward an answer for sounding complete when the source does not contain the answer. One of the test cases should require the consumer to acknowledge missing information. A useful content guide should make the supported facts easier to find, not encourage confident invention.

The marketing AI evaluation guide explains how to separate answer quality from critical failures. Apply the same discipline here: the file is useful if it improves a specific retrieval task with accurate, maintainable content.

What belongs in the maintenance checklist?

Treat this as a small publishing surface with an owner, not a permanent badge of AI readiness.

Review the file when products, policies, URLs or commercial details change. If no consumer uses it and maintenance creates drift, a smaller guide or no guide can be a better decision. The priority remains a useful, accurate website that people can navigate and trust.