Skip to content

docs: serve llms.txt and per-page markdown for LLM consumers#188

Draft
Lutherwaves wants to merge 1 commit into
mainfrom
docs/llm-friendly-serving
Draft

docs: serve llms.txt and per-page markdown for LLM consumers#188
Lutherwaves wants to merge 1 commit into
mainfrom
docs/llm-friendly-serving

Conversation

@Lutherwaves
Copy link
Copy Markdown
Contributor

Summary

Makes the docs site consumable by LLMs and agents as text, not rendered HTML — addresses the pushback on shipping more HTML chrome before improving how the existing markdown is served.

Three entry points after this PR:

  • /llms.txt — short, curated index per the llmstxt.org spec. Lists each guide with its URL pointing at the raw .md.
  • /llms-full.txt — entire site concatenated as plain markdown (~2.4k lines today). Single-file ingest for embedding pipelines / RAG.
  • "View source" button next to "Edit this page" on every page → links to the raw .md on GitHub for one-off agent access.

Changes

  • requirements-docs.txt — add mkdocs-llmstxt==0.5.0 (pure Python, no new native deps; Cairo etc. are not required for this plugin)
  • mkdocs.yml
    • add content.action.view Material feature
    • add plugins: block with search (re-declared because adding the section disables the default) and llmstxt configured with curated sections matching the existing nav

Verification

$ mkdocs build --strict
INFO    -  Documentation built in 0.64 seconds

$ ls site/llms*.txt
-rw-r--r-- … 99221 … site/llms-full.txt
-rw-r--r-- …   584 … site/llms.txt

site/llms.txt:

# magic

> Common building blocks for Go microservices, so your service code can focus on business logic.

## Overview
- [Home](https://tink3rlabs.github.io/magic/index.md)

## Guides
- [Getting Started](https://tink3rlabs.github.io/magic/getting-started/index.md)
- [Search (Lucene)](https://tink3rlabs.github.io/magic/lucene/index.md)
- [Storage Adapters](https://tink3rlabs.github.io/magic/storage/index.md)
- [Observability](https://tink3rlabs.github.io/magic/observability/index.md)

## Contributing
- [Contributing](https://tink3rlabs.github.io/magic/contributing/index.md)

Test plan

  • CI's Docs workflow builds without errors
  • After merge: https://tink3rlabs.github.io/magic/latest/llms.txt resolves
  • After merge: https://tink3rlabs.github.io/magic/latest/llms-full.txt resolves
  • After merge: a page header shows the new "View source" eye icon next to the edit pencil

Out of scope (future)

  • Per-page lead summaries to improve llms.txt entries (content work, not plumbing)
  • Standardizing code-block titles for better chunk metadata
  • A hand-curated llms.txt override (the auto-generated one is good enough for now)

🤖 Generated with Claude Code

Adds the mkdocs-llmstxt plugin and Material's content.action.view
feature so the site exposes three LLM-friendly entry points:

- /llms.txt — curated index per llmstxt.org
- /llms-full.txt — entire site concatenated as plain markdown
- "View source" button per page → raw .md on GitHub

Lets agents and crawlers consume the docs as text rather than
scraping rendered HTML.
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 13, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 5370c34d-25cc-4507-987c-9eae1863ac03

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch docs/llm-friendly-serving

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant