Add combined compacted llms-full.txt; make llms.txt a product directory

- New build_llms_full.py aggregates every product's source into one
  compacted, well-structured /llms-full.txt (navigation boilerplate stripped;
  signatures, summaries, and member/parameter tables kept). ~28 MB / ~3 MB gz.
- Site-root llms.txt is now a per-product directory (the format previously used
  by llms-full.txt) with a pointer to the combined file.
- Home build no longer emits llms-full.txt; build_llms_full.py owns the
  bucket-root /llms-full.txt, built and deployed by deploy_all and the content
  refresher (refresh_search_index.yml -> "Refresh Root Aggregates").
- All page links in llms.txt / llms-full.txt now target the Markdown (.md)
  rendition instead of the HTML page, for LLM/agent consumption.
- Docs updated: README, CLAUDE.md, CHANGELOG.md.

Author: vladimir-litvinchik

.github/workflows/deploy_all.yml

@@ -58,8 +58,9 @@ jobs:
       environment: ${{ github.event_name == 'workflow_dispatch' && inputs.environment || (github.ref_name == 'production' && 'production' || 'staging') }}
     secrets: inherit
 
-  # Rebuild the root /search-index.json (the 404 live search source) from every product's
-  # source front matter — no per-product build artifacts needed. Same job runs standalone in
+  # Rebuild the bucket-root aggregates from every product's SOURCE — no per-product build
+  # artifacts needed: the 404 live-search index (/search-index.json) and the combined,
+  # compacted whole-reference file (/llms-full.txt). Same job runs standalone in
   # refresh_search_index.yml on content pushes.
   search-index:
     if: ${{ !cancelled() }}
@@ -80,11 +81,25 @@ jobs:
           AWS_ACCESS_KEY_ID: ${{ secrets.ACCESS_KEY }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.SECRET_ACCESS }}
 
+      - name: Build llms-full.txt from source
+        run: |
+          BASE=${{ env.ENVIRONMENT == 'production' && 'https://reference.groupdocs.com' || 'https://reference2.groupdocs.com' }}
+          python3 build_llms_full.py --source content/sites/groupdocs --base-url "$BASE" --out llms-full.txt
+
+      - name: Deploy llms-full.txt to bucket root
+        run: |
+          BUCKET=${{ env.ENVIRONMENT == 'production' && 'reference.groupdocs.com' || 'reference2.groupdocs.com' }}
+          aws s3 cp llms-full.txt "s3://$BUCKET/llms-full.txt" \
+            --content-type "text/plain; charset=utf-8" --region us-west-2
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.ACCESS_KEY }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.SECRET_ACCESS }}
+
       - name: Invalidate CloudFront
         uses: chetan/invalidate-cloudfront-action@v2
         env:
           DISTRIBUTION: ${{ env.ENVIRONMENT == 'production' && secrets.AWS_DISTRIBUTION_PROD || secrets.AWS_DISTRIBUTION }}
-          PATHS: '/search-index.json'
+          PATHS: '/search-index.json /llms-full.txt'
           AWS_REGION: 'us-west-2'
           AWS_ACCESS_KEY_ID: ${{ secrets.ACCESS_KEY }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.SECRET_ACCESS }}

.github/workflows/refresh_search_index.yml

@@ -1,8 +1,9 @@
-name: Refresh Search Index
+name: Refresh Root Aggregates
 
-# Lightweight: rebuild the root /search-index.json (the 404 live search source) from product
-# source front matter — no Hugo build. Runs on any content push so the index stays current
-# between full Deploy All runs. Branch mapping: production -> production, main -> staging.
+# Lightweight: rebuild the bucket-root aggregates from product SOURCE — no Hugo build:
+# /search-index.json (the 404 live search source) and /llms-full.txt (the combined, compacted
+# whole-reference file). Runs on any content push so both stay current between full Deploy All
+# runs. Branch mapping: production -> production, main -> staging.
 on:
   workflow_dispatch:
     inputs:
@@ -44,11 +45,25 @@ jobs:
           AWS_ACCESS_KEY_ID: ${{ secrets.ACCESS_KEY }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.SECRET_ACCESS }}
 
+      - name: Build llms-full.txt from source
+        run: |
+          BASE=${{ env.ENVIRONMENT == 'production' && 'https://reference.groupdocs.com' || 'https://reference2.groupdocs.com' }}
+          python3 build_llms_full.py --source content/sites/groupdocs --base-url "$BASE" --out llms-full.txt
+
+      - name: Deploy llms-full.txt to bucket root
+        run: |
+          BUCKET=${{ env.ENVIRONMENT == 'production' && 'reference.groupdocs.com' || 'reference2.groupdocs.com' }}
+          aws s3 cp llms-full.txt "s3://$BUCKET/llms-full.txt" \
+            --content-type "text/plain; charset=utf-8" --region us-west-2
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.ACCESS_KEY }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.SECRET_ACCESS }}
+
       - name: Invalidate CloudFront
         uses: chetan/invalidate-cloudfront-action@v2
         env:
           DISTRIBUTION: ${{ env.ENVIRONMENT == 'production' && secrets.AWS_DISTRIBUTION_PROD || secrets.AWS_DISTRIBUTION }}
-          PATHS: '/search-index.json'
+          PATHS: '/search-index.json /llms-full.txt'
           AWS_REGION: 'us-west-2'
           AWS_ACCESS_KEY_ID: ${{ secrets.ACCESS_KEY }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.SECRET_ACCESS }}

CHANGELOG.md

@@ -10,6 +10,11 @@ tags), so changes accumulate under **[Unreleased]**.
 ## [Unreleased]
 
 ### Added
+- **Combined, compacted `llms-full.txt`** at the site root — the entire API reference for all 15
+  products in one well-structured file. Built from source by `build_llms_full.py` (navigation
+  boilerplate stripped: `See Also` footers, `Learn more` blocks, internal link URLs, front matter,
+  heading anchors, Java separators; signatures, summaries, and member/parameter tables kept).
+  ~28 MB (~3 MB gzipped). Deployed to the bucket root by `deploy_all` + the content-push refresher.
 - **Live search on the 404 page** — type-ahead over a combined `/search-index.json` (all 15 products,
   ~37k symbols): kind badges, namespace paths, platform pills, keyboard navigation.
 - **Cross-product search index** `/search-index.json`, built from product source front matter by
@@ -31,8 +36,15 @@ tags), so changes accumulate under **[Unreleased]**.
   `deploy_home.yml`), Hugo 0.101.0, AWS S3 + CloudFront.
 - Home page: product grid driven by `data/products.toml`; **AI Agents & LLM section** refocused on this
   site's machine-readable indexes (`/llms.txt`, `/llms-full.txt`, `.md`) with clearer FontAwesome icons.
-- Site-root `llms.txt`/`llms-full.txt` and home `index.md` emit **absolute URLs** (host baseURL per env);
-  site-root `llms-full.txt` is a directory of per-product `llms-full.txt`.
+- Site-root `llms.txt`/`llms-full.txt` and home `index.md` emit **absolute URLs** (host baseURL per env).
+- **All page links in `llms.txt` / `llms-full.txt` now point to the Markdown (`.md`) rendition, not the
+  HTML page** — these files are for LLMs/agents (e.g. `…/annotator/text/` → `…/annotator/text.md`).
+  Applies to the combined `/llms-full.txt` (`Path:` lines) and the per-product Hugo `llms.txt`/`llms-full.txt`.
+- **Site-root `llms.txt` is now a per-product directory** (full-reference / section-index / Markdown
+  links per product, plus a pointer to the combined `/llms-full.txt`) — the format previously used by
+  `llms-full.txt`. The home build no longer emits `llms-full.txt`; the source-built combined file owns
+  `/llms-full.txt`. `refresh_search_index.yml` (now "Refresh Root Aggregates") rebuilds both
+  `/search-index.json` and `/llms-full.txt` on content pushes.
 
 ### Fixed
 - Family-page "Browse API reference" links now include the product path prefix.

CLAUDE.md

@@ -29,8 +29,10 @@ obvious from the code.
     → `hugo deploy` → CloudFront invalidate**.
   - `deploy_<product>.yml` (×15) — triggers on `content/sites/groupdocs/<product>/**` push; calls the reusable one.
   - `deploy_home.yml` — `content/sites/groupdocs/home/**`; invalidates `/index.html /index.md /llms.txt /llms-full.txt /404.html`; `maxDeletes 0`.
-  - `deploy_all.yml` — `themes/**` push or manual; home → products matrix → **`search-index`** job (rebuilds `/search-index.json` from source).
-  - `refresh_search_index.yml` — `content/sites/groupdocs/**` push or manual; rebuilds `/search-index.json` from source (lightweight, no Hugo), concurrency-coalesced.
+  - `deploy_all.yml` — `themes/**` push or manual; home → products matrix → **aggregates** job (`search-index`)
+    that rebuilds the bucket-root `/search-index.json` **and** `/llms-full.txt` from source.
+  - `refresh_search_index.yml` ("Refresh Root Aggregates") — `content/sites/groupdocs/**` push or manual;
+    rebuilds `/search-index.json` **and** `/llms-full.txt` from source (lightweight, no Hugo), concurrency-coalesced.
 - Secrets: `ACCESS_KEY`/`SECRET_ACCESS` (S3), `AWS_DISTRIBUTION_PROD` (prod) / `AWS_DISTRIBUTION` (staging) CloudFront.
 
 ## Content model
@@ -58,8 +60,15 @@ obvious from the code.
   **absolute links**, **ugly URLs**. Family page `.md` is served at **`/<product>.md`** (bucket root), not `/<product>/index.md`.
 - `partials/md/abs-content.txt`: emits `.RawContent`, absolutizes root-relative links, strips a leading
   front-matter block for empty-body pages (CRLF-tolerant).
-- llms: `index.llmstxt.txt` (home lists products from `data/products.toml` with `.md` links),
-  `index.llmsfull.txt` (product build inlines all pages; site-root = directory of per-product `llms-full.txt`).
+- llms (per-build, Hugo): `index.llmstxt.txt` — **site-root** `llms.txt` is a per-product **directory**
+  (full-reference / section-index / Markdown links per product, from `data/products.toml`); a product
+  build's `llms.txt` lists its platforms/namespaces. `index.llmsfull.txt` — a **product** build inlines
+  all its pages into `/<product>/llms-full.txt`. The **site-root** build no longer emits `llms-full.txt`.
+- **Combined `/llms-full.txt`** (bucket root): the entire reference for all 15 products in one compacted,
+  well-structured file, built from source by `build_llms_full.py` (no Hugo — each product is a separate
+  build, so Hugo can't aggregate). It strips nav boilerplate (`See Also`, `Learn more`, internal link
+  URLs, front matter, anchors, Java separators) and keeps signatures + summaries + member/parameter
+  tables. Produced & deployed by the `deploy_all` aggregates job and `refresh_search_index.yml`.
 - **Search index** `/search-index.json` (bucket root, ~37k entries, all products) powers the **404 live search**.
   Built by `build_search_index.py --source content/sites/groupdocs` (parses front matter `title`+`url`, no Hugo).
 - Serving: `.md`/`.txt`/`.html` get `charset=utf-8` via `[[deployment.matchers]]`; CloudFront auto-gzips.
@@ -69,6 +78,8 @@ obvious from the code.
 - `resolve_md_links.py <dir> --base-url <BASE>` — resolve relative `.md` links to absolute (per page URL);
   `surrogateescape` for non-UTF-8 bytes. Must run **before** the ugly-URL rename.
 - `build_search_index.py --source content/sites/groupdocs --out search-index.json` — combined index (source mode).
+- `build_llms_full.py --source content/sites/groupdocs --base-url <BASE> --out llms-full.txt` — combined,
+  compacted whole-reference file (source mode; `--only a,b` limits products for fast local preview).
 - `build-local.sh [products…]` — builds home + listed products into `./public-local/` (runs resolver + ugly rename +
   family-md-to-root + search index). Default product: annotation.
 - `serve-local.py [port]` — static server for `./public-local` that sends `charset=utf-8` for text (plain

README.md

@@ -53,6 +53,13 @@ Alongside HTML, every page emits a **Markdown (`.md`)** rendition, and each buil
 consumption. Configured per product in `config/sites/groupdocs/<product>/_default/config.toml`
 (`[outputFormats.MD|LLMSTXT|LLMSFULL]`, `[outputs]`).
 
+At the **site root**, `llms.txt` is a **per-product directory** (each product's full-reference,
+section-index, and Markdown links), and `/llms-full.txt` is the **entire API reference for all 15
+products in one compacted file**. Because each product is a separate Hugo build, Hugo can't
+aggregate them — so [`build_llms_full.py`](build_llms_full.py) builds `/llms-full.txt` from source
+(stripping navigation boilerplate, keeping signatures + summaries + member/parameter tables) and CI
+uploads it to the bucket root (`deploy_all` + `refresh_search_index.yml`).
+
 The `.md` files are post-processed so they're clean and portable:
 
 1. **No front matter, real Markdown** — `single.md` / `list.md` emit `.RawContent` (Hugo strips

build-local.sh

@@ -43,6 +43,11 @@ done
 # Combined cross-product search index for the 404 live search (built from the merged tree).
 python build_search_index.py "$OUT" --out "$OUT/search-index.json"
 
+# Combined, compacted llms-full.txt (whole API reference, built from source). --only limits it
+# to the products built locally so the preview is quick; production builds all 15.
+python build_llms_full.py --source content/sites/groupdocs --base-url "http://localhost:$PORT" \
+  --only "$(IFS=,; echo "${PRODUCTS[*]}")" --out "$OUT/llms-full.txt"
+
 echo
 echo "Built ${#PRODUCTS[@]} product(s) + home into ./$OUT"
 echo "Serve with:  python serve-local.py $PORT      # UTF-8 charset, matches production"