feat(seo): resolve Search Console issues + AI agent-readiness#2135
Merged
Conversation
- Absolute canonical/OG/breadcrumb/JSON-LD URLs across routes; Dataset creator+license - Category-aware entry schema; strip relative-href links from rendered content (404 source) - robots Disallow /api,/data,/downloads,/_next + Content-Signal; non-prod host noindex - Per-entry + per-category sitemap lastmod; reclaim /<category> as indexable hubs - browse invalid-category 301; /search->/browse and /jobs/post canonicals Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- /.well-known/api-catalog (RFC 9727), /.well-known/mcp/server-card.json (SEP-1649) - /.well-known/agent-skills/index.json (checksummed skill packages) - RFC 8288 Link headers advertising the above on HTML responses - WebMCP provider exposing directory search to in-browser agents - docs/agent-discovery.md incl. DNS-AID ops steps Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- robots Disallow + Content-Signal, host-aware noindex, Link header - sitemap excludes disallowed /data, adds category hubs + per-category lastmod Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dosubot
Bot
added
the
size:XL
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (7)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (5)
📝 WalkthroughWalkthroughAdds an in-page WebMCP tool and multiple well-known discovery endpoints, centralizes crawler disallow rules and a Content-Signal, makes security headers request-aware (non-prod noindex + discovery Link), introduces category/tag/platform hub routes with enriched SEO/JSON-LD, updates sitemap lastmod handling and static-path exclusions, and aligns tests and docs. ChangesAI Agent Discovery Platform
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested labels
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Note Gittensory Gate skippedPR closed before full evaluation. No late first comment was created.
💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers. |
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
🔍 Maintainer advisory reviewReviewed 38 changed file(s) — two independent AI reviewers. This is an advisory review — it does not merge or close the PR. Suggested action: 🛠️ Request changes. (reviewers split: merge / request changes) Address the suggestions below before merging. Reviewer A · Suggestions
Worth double-checking
Reviewer B · Suggestions
Worth double-checking
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/web/src/routes/feeds.tsx (1)
11-28:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winAdd missing
og:urlmeta tag to feeds route.The
/feedsroute's<head>metadata hascanonicalwithabsoluteUrl()but is missing theog:urlmeta tag. All peer routes in this cohort (ecosystem, quality, jobs.index, jobs.$slug) include both. Per the PR objectives, OG URLs should be absolute across routes.🔗 Proposed fix to add og:url
meta: [ { title: "Feeds & subscriptions — HeyClaude" }, { name: "description", content: "Subscribe to HeyClaude registry updates via RSS, Atom, or email. Follow the whole registry, a single category, or a changelog stream.", }, { property: "og:title", content: "Feeds & subscriptions — HeyClaude" }, { property: "og:description", content: "RSS, Atom, and email subscriptions for the HeyClaude registry.", }, + { property: "og:url", content: absoluteUrl("/feeds") }, ],🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/web/src/routes/feeds.tsx` around lines 11 - 28, The meta array in the feeds route is missing an Open Graph URL entry; add a meta object { property: "og:url", content: absoluteUrl("/feeds") } to the meta array in apps/web/src/routes/feeds.tsx (near the existing canonical/og:title/og:description entries) so the OG URL is absolute and consistent with the other routes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/web/src/routes/`[.]well-known.agent-skills.index[.]json.ts:
- Around line 8-16: The skills[] construction currently sets url to the entry
HTML page; change it to use the artifact download URL so clients can fetch the
bytes that match sha256: in the skills mapping (where skills is built from
ENTRIES.filter(...).map(...)) replace the url value that uses
absoluteUrl(`/entry/${e.category}/${e.slug}`) with the checked artifact URL
(e.downloadUrl or absoluteUrl(e.downloadUrl) if downloadUrl is relative)
ensuring the url and sha256 now point at the same downloadable artifact.
In `@apps/web/src/routes/validators.tsx`:
- Around line 54-55: The JSON-LD emitter in validators.tsx is currently always
setting datePublished and dateModified to String(atlasRegistry.generatedAt ||
"").slice(0,10), which emits empty/invalid date strings when
atlasRegistry.generatedAt is missing; change the code that builds the JSON-LD
object (the place that sets datePublished and dateModified) to only include
those properties when atlasRegistry.generatedAt is truthy (e.g., compute const
generatedDate = atlasRegistry.generatedAt ?
String(atlasRegistry.generatedAt).slice(0,10) : undefined and either
conditionally spread { ...(generatedDate && { datePublished: generatedDate,
dateModified: generatedDate }) } into the object or add the properties inside an
if block), referencing the datePublished/dateModified fields and
atlasRegistry.generatedAt to locate where to apply the conditional inclusion.
In `@docs/agent-discovery.md`:
- Line 36: Change the opening fenced code block for the DNS SVCB example from
``` to ```dns so the block has a language identifier (e.g., use ```dns) to
satisfy markdownlint MD040; no other content changes required.
---
Outside diff comments:
In `@apps/web/src/routes/feeds.tsx`:
- Around line 11-28: The meta array in the feeds route is missing an Open Graph
URL entry; add a meta object { property: "og:url", content:
absoluteUrl("/feeds") } to the meta array in apps/web/src/routes/feeds.tsx (near
the existing canonical/og:title/og:description entries) so the OG URL is
absolute and consistent with the other routes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 9d2d98d6-a61c-4205-9f32-a97b0dd1ac2c
📒 Files selected for processing (31)
apps/web/src/components/webmcp-provider.tsxapps/web/src/lib/detail-assembly.tsapps/web/src/lib/robots-policy.tsapps/web/src/lib/security-headers.tsapps/web/src/routes/$category.tsxapps/web/src/routes/[.]well-known.agent-skills.index[.]json.tsapps/web/src/routes/[.]well-known.api-catalog.tsapps/web/src/routes/[.]well-known.mcp.server-card[.]json.tsapps/web/src/routes/__root.tsxapps/web/src/routes/best.$slug.tsxapps/web/src/routes/best.tsxapps/web/src/routes/browse.tsxapps/web/src/routes/changelog.tsxapps/web/src/routes/compare.tsxapps/web/src/routes/ecosystem.tsxapps/web/src/routes/entry.$category.$slug.tsxapps/web/src/routes/feeds.tsxapps/web/src/routes/index.tsxapps/web/src/routes/jobs.$slug.tsxapps/web/src/routes/jobs.index.tsxapps/web/src/routes/jobs.post.tsxapps/web/src/routes/quality.tsxapps/web/src/routes/search.tsapps/web/src/routes/sitemap[.]xml.tsapps/web/src/routes/subscriptions.tsxapps/web/src/routes/validators.tsxapps/web/src/server.tsdocs/agent-discovery.mdtests/atlas-production-data.test.tstests/crawler-policy.test.tstests/sitemap-policy.test.ts
…copy - /tags + /tags/:tag topic hubs (ItemList+breadcrumb JSON-LD); entry tag chips now link out - /for + /for/:platform landing pages per platform (ItemList+FAQ+breadcrumb), reusing search() - HowTo JSON-LD for guides (steps from headings) - Unique, per-entry fallback copy for bodyless entries (category/tags/platforms) - Sitemap: tag hubs + platform pages Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ta/llms - MCP server-card version sourced from the mcp-server integration (synced to packages/mcp) - Drop deprecated /data/llms llms-url tolerance; generators already use /api/registry/.../llms Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dosubot
Bot
added
size:XXL
- agent-skills index: url points at the download artifact matching sha256 (not entry page) - validators Dataset: omit datePublished/dateModified when generatedAt is missing - feeds: add absolute og:url - docs: language hint on DNS-AID fenced block Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
apps/web/src/lib/tags.ts (1)
14-30: ⚡ Quick winTag name may vary if entries use different casings for the same slug.
Line 25 sets
name: tagusing the first encountered raw tag string for each slug. If different entries use different casings (e.g.,"AI"vs."ai"), the group name will be whichever appears first, potentially causing display inconsistency.Consider normalizing tag names upstream in the data layer, or selecting a canonical name (e.g., the most common casing or title-case).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/web/src/lib/tags.ts` around lines 14 - 30, getAllTagGroups currently uses the first raw tag string as the group's name which can vary by casing; update getAllTagGroups to choose a canonical name per slug (e.g., compute the most frequent raw tag casing among entries for that slug or apply a deterministic normalization like title-case) instead of using the first seen tag; reference getAllTagGroups and tagSlug when locating the logic that builds TagGroup objects and modify the branch that sets name: tag so it aggregates candidate names (or normalizes) and assigns the chosen canonical name before caching the TagGroup list.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/web/src/routes/entry`.$category.$slug.tsx:
- Around line 551-555: The Link rendering currently uses tagSlug(t) directly
which can return an empty string and produce broken /tags/$tag links; update the
rendering around the Link that uses tagSlug(t) so you compute const slug =
tagSlug(t) and only render the <Link ... to="/tags/$tag" params={{ tag: slug }}
...> when slug is non-empty, otherwise render a non-interactive chip (e.g., a
<span> or <div> with the same classes and content) so empty slugs do not produce
navigable/broken links; apply the same guard to the other identical instance
that uses tagSlug(t).
- Around line 116-129: The guideHowTo function currently maps all headings into
HowTo steps, but the comment specifies only H2/H3 should be emitted; update the
step construction to first filter (e.headings ?? []) for headings with level ===
2 or level === 3 and then map that filtered array to the HowToStep objects
(e.g., replace the current step: (e.headings ?? []).map(...) with (e.headings ??
[]).filter(h => h.level === 2 || h.level === 3).map(...)); apply the same
filter-and-map change to the other HowTo step serialization in this file so only
H2/H3 headings are included.
In `@scripts/validate-raycast-feed.mjs`:
- Around line 284-286: The regex for validLlmsUrl is too permissive (it allows
extra path segments via (?:/.*)?); update the validation so validLlmsUrl
enforces the exact endpoint shape by replacing the pattern with one that anchors
the end, e.g. use /^\/api\/registry\/entries\/[^/]+\/[^/]+\/llms\/?$/ (allowing
optional trailing slash) against llmsUrl in the same check that sets
validLlmsUrl and leave the subsequent fail(...) behavior unchanged.
---
Nitpick comments:
In `@apps/web/src/lib/tags.ts`:
- Around line 14-30: getAllTagGroups currently uses the first raw tag string as
the group's name which can vary by casing; update getAllTagGroups to choose a
canonical name per slug (e.g., compute the most frequent raw tag casing among
entries for that slug or apply a deterministic normalization like title-case)
instead of using the first seen tag; reference getAllTagGroups and tagSlug when
locating the logic that builds TagGroup objects and modify the branch that sets
name: tag so it aggregates candidate names (or normalizes) and assigns the
chosen canonical name before caching the TagGroup list.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: ad1c7ee6-2de8-47ff-8bc8-f60b474517bc
📒 Files selected for processing (10)
apps/web/src/lib/tags.tsapps/web/src/routes/[.]well-known.mcp.server-card[.]json.tsapps/web/src/routes/entry.$category.$slug.tsxapps/web/src/routes/for.$platform.tsxapps/web/src/routes/for.index.tsxapps/web/src/routes/sitemap[.]xml.tsapps/web/src/routes/tags.$tag.tsxapps/web/src/routes/tags.index.tsxintegrations/raycast/test/feed.test.tsscripts/validate-raycast-feed.mjs
🚧 Files skipped from review as they are similar to previous changes (1)
- apps/web/src/routes/[.]well-known.mcp.server-card[.]json.ts
- entry tags: render a static chip when the tag slugifies to empty (no broken /tags link) - guide HowTo: emit steps from H2/H3 headings only (filter by depth), shared with the guard - raycast feed validator: anchor the llms-url regex to the exact endpoint shape - tags: pick the most-frequent raw casing as a tag's canonical display name Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Resolves every Google Search Console finding for heyclau.de and adds an SEO growth + AI-agent-discovery layer.
GSC fixes
Invalid URL in item): absolute breadcrumb/canonical/OG/JSON-LD URLs across all routes.creator+licenseadded.robots.txtdisallows/api,/data,/downloads,/_next; disallowed URLs removed from sitemap./search→/browse301,/jobs/postcanonical.X-Robots-Tag: noindexfor non-prod hosts.Growth
/<category>(ItemList/FAQ/Breadcrumb JSON-LD)./tags+/tags/:tag(entry tag chips now link out)./for+/for/:platform(e.g. "Claude resources for Cursor").lastmod; tag/platform pages in sitemap.AI agent readiness
/.well-known/api-catalog(RFC 9727),/.well-known/mcp/server-card.json(SEP-1649, version synced to packages/mcp),/.well-known/agent-skills/index.json(checksummed skills).Linkheaders; fully-openContent-Signal; WebMCP provider..devia the current registrar (documented indocs/agent-discovery.md).Housekeeping
/data/llms/*.txtscheme (generators already use/api/registry/.../llms).Validation
pnpm type-check,pnpm test(703),pnpm validate:raycast-feed(922), raycastnode --test(49),pnpm validate:openapi,pnpm build,git diff --check— all green. Reviewed via a multi-angle finder→verify pass; fixes applied (orphan-anchor a11y, duplicate Content-Signal, ranking reuse, canonical builder alignment).🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Tests