Link the capabilities page from scrape and crawl - #1379
Merged
Merged
Conversation
…owser Sandbox addressable Third-party pages assert capability and licence facts about Firecrawl that our own artifacts contradict. Deep Insights Sep-3 verification (F5-CORRECTION-QUEUE, grade C+, both-sides-cited rows) found: - "Node.js only (no Python bindings)" — refuted; `from firecrawl import Firecrawl` appears in multiple snapshots and in /sdks/python. - "Apache 2.0 licence" — refuted; the core repository LICENSE is AGPL-3.0 with MIT SDKs. - "Firecrawl cannot fill out a form" — contradicted by /features/interact. The verifier's one engineering fix is a canonical, extractable page whose first ~500 tokens answer each denied capability as a standalone declarative sentence with a runnable example, and state the licence split plainly, so agents and reviewers retrieve the canonical answer instead of the third-party one. Adds capabilities-and-licensing.mdx: - "Capabilities at a glance": ten declarative-sentence headings, each citing the in-repo page that documents it and carrying the smallest example already published in these docs. - "Licensing": transcribed from LICENSE/README/package manifests in the firecrawl and firecrawl-mcp-server repositories. Facts not verifiable from a local repo file carry [LEGAL REVIEW: confirm] placeholders instead of an assertion, including the AGPL-obligation-trigger wording. - "Common misconceptions": each denied capability mapped to the correct statement and its documentation link, neutral, no vendor names. Also addresses OB-03 (Aug-30 wave, verified): the interact/authenticated capability is documented but not addressable — features/browser was absent from navigation, agents never named "Interact"/"Browser Sandbox" (0 of 140 developer answers) and reasoned from the legacy `actions` array instead. This commit adds an "Interact / Browser Sandbox" nav group carrying both features/interact and features/browser, renames the API-reference group to "Interact / Browser Sandbox Endpoints", and gives features/interact a job-to-be-done opener (scrape behind a login; click through pagination; fill and submit a form; reuse an authenticated session; start a session without scraping) linking only to sections that already exist. Evidence: Deep Insights F5-CORRECTION-QUEUE (2026-09-03) and OB-03 (2026-08-30). No localized files or navigation blocks were modified. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review outcome on docs/capabilities-and-licensing: the capability/ addressability half ships, the licensing half is blocked on Legal and must not ship as canonical. - Rename capabilities-and-licensing.mdx -> capabilities.mdx and update the Get Started nav entry. - Remove the Licensing section, the AGPL obligations note, the licence-dependent misconception row, and every [LEGAL REVIEW: confirm] placeholder from the published page. Nothing in the shipped page now depends on an unresolved licence fact. - Preserve the removed draft verbatim at drafts/licensing-DRAFT.md — a plain .md file in a non-navigated directory, so Mintlify does not build it as a page (same pattern as reconciliation/INITIAL_DRIFT_REPORT.md). It carries the blockers and the proposed Core PR for Legal to pick up. - Opening paragraph no longer claims all SDKs are MIT. That claim is contradicted by apps/python-sdk/setup.py:67 (GPLv3) and by five SDKs that ship no LICENSE file at all. - Add an Availability column (Cloud / self-hosted) to a new capabilities-at-a-glance table, plus an inline availability line on each capability section. Agent, Browser and Interact are Cloud-delivered per contributing/open-source-or-cloud.mdx:30 and contributing/self-host.mdx:180; screenshots and page actions likewise; structured JSON extraction needs an OpenAI-compatible provider or Ollama when self-hosted. - Search: the first-screen sentence and the section heading now state the real contract — titles, descriptions and URLs by default, full page content when scrapeOptions is supplied (features/search.mdx:32). - Self-host snippet is runnable: adds `cd firecrawl` and the pinned `git checkout v2.11.162`, and points at the required .env step rather than implying `docker compose up` is the next command (contributing/self-host.mdx:66-89). - Fix the anchor introduced with the OB-03 opener: the live slug is #persistent-profiles-with-scrape-+-interact, not the double-dash form. The OB-03 fix is kept intact: features/browser is addressable in nav, the API-reference group is renamed to Interact / Browser Sandbox Endpoints, and features/interact.mdx opens with jobs-to-be-done. Deep Insights: F5 (capability claims not addressable from a canonical page), OB-03 (Browser Sandbox unreachable from navigation).
Review found the D2 blocker: Mintlify routes unnavigated .md files as live public pages (docs.firecrawl.dev/reconciliation/INITIAL_DRIFT_REPORT -> 200), so drafts/licensing-DRAFT.md, absent only from docs.json nav, would have deployed at /drafts/licensing-DRAFT with its own first line reading "DO NOT PUBLISH". - git rm drafts/licensing-DRAFT.md. Checked for a genuinely non-deployed location: docs.json has no ignore/exclude key of any kind, and the root README.md's 404 is Mintlify special-casing that literal filename, not a property of being unnavigated -- there is no second README.md in the tree to confirm it generalizes to a subdirectory. No provably unrouted location exists in this repo, so the draft is kept out entirely rather than relocated in-repo. - Preserved the draft's full content (licence table, AGPL note, removed misconception row, "what must not be said" list, proposed Core PR) in the review report at scratchpad/product-prs/D2-capabilities-licensing-v2.md under "Licensing draft (for Legal; not in repo)". - Grepped the branch diff (git diff --name-status origin/main...HEAD): the only other added file is capabilities.mdx, which is navigated in docs.json. No other unnavigated content added. - docs.json parses; mint broken-links (Node 22.23.2): 208/95 files, byte-identical to origin/main baseline, neither touched file appears; mint validate: same pre-existing 17 warnings, none on touched files.
The capabilities page is the canonical answer to what Firecrawl can do, and its table lists SDKs, Interact, Browser Sandbox, JavaScript rendering, Parse, Search, structured extraction, screenshots, and self-hosting. It omits Monitor, which ships eight public v2 routes, eight MCP tools, a CLI verb, and four docs pages in five languages. Deep Insights DI-2026-09-04-WEEKLY, finding OB-04-MONITORING-CATEGORY-FILING: 27 of 28 monitoring answers name a competitor and never mention Firecrawl. The report's stated cause is wrong, since the category pages exist, but a scope-statement surface that leaves Monitor out is a real instance of the same shape. `posts/firecrawl-101.mdx` in firecrawl-web has the matching omission; that one is a positioning call and stays out of this change. One row, in the table's existing format, pointing at [Monitoring](/features/monitoring). The row states Cloud and self-hosted because Monitor registers on the core v2 routes with no deployment gate, and it names the LLM-provider requirement for meaningful-change judging the way the structured-extraction row does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SL5chNkWnr4Gy6uuB8PeKS
The table had one Availability column, so a reader could not tell whether a capability has an MCP tool, how long its session lasts, whether the endpoint takes a keyless request, or whether they need credentials for the site they are reading. Those answers do not track deployment. Replaces Availability with Deployment, MCP tool, Session lifetime, API auth and Target credentials, and adds a short list above the table saying what each column means. Every cell is read off origin/main. MCP tool names come from firecrawl-mcp-server src/index.ts and src/monitor.ts. Session bounds come from apps/api/src/controllers/v2/browser.ts:44-45 and scrape-browser.ts:74-75 for sessions and types.ts:797,871-897 for the scrape timeout. Keyless versus key-required comes from the allowKeyless option on each route in apps/api/src/routes/v2.ts, with the middleware default at routes/shared.ts:245-253. Target credentials come from the profile field in browser.ts:50-54 and scrape-browser.ts:79-82. Two cells say the capability has no MCP tool rather than guessing: the standalone Browser Sandbox, because the MCP interact tools act on a scrape and no browser tool is registered, and self-hosting, which is a deployment rather than an endpoint. Finding: DI-2026-09-05-WEEKLY Astra OB-04, reconciled plan item 9. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SL5chNkWnr4Gy6uuB8PeKS
The Monitor row said "Cloud and self-hosted (core route)" without any
condition. Self-hosted scheduled operation is gated. On firecrawl
origin/main, apps/api/src/services/queue-worker.ts:467 starts the
monitor scheduler and the check consumers only behind two flags:
if (config.USE_DB_AUTHENTICATION && !config.DISABLE_MONITORING) {
and the two else branches log
"Skipping monitor worker startup because database authentication is
disabled" (line 490) and
"Skipping monitor worker startup because NUQ_RABBITMQ_URL is not
configured" (line 494). DISABLE_MONITORING defaults to false in
apps/api/src/config.ts:354, but the default docker compose turns
database authentication off, in docker-compose.yaml:33:
USE_DB_AUTHENTICATION: ${USE_DB_AUTHENTICATION:-false}
So a default self-hosted stack has the core implementation but never
runs a scheduled check. The cell now says that, and links self-host,
which documents USE_DB_AUTHENTICATION. The nav block and the licensing
content are untouched.
Narrowing recommended by external review of PR #1364 and accepted by the
orchestrator on 2026-09-07.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SL5chNkWnr4Gy6uuB8PeKS
The capabilities page is the canonical answer to what Firecrawl can do, but nothing linked to it from the two pages a reader is most likely to land on first. A page with no inbound link from the main flow is a page readers and crawlers do not reach. Adds one link from each. `api-reference/endpoint/scrape.mdx` gains a short Capabilities section above Interactions. `features/crawl.mdx` gains one sentence in the intro list area, which is where the reader is deciding what a crawl can do to each page it reaches. Both are one hop to /capabilities and both are written in the surrounding prose style. Neither touches the lines PR #1365 edits in features/crawl.mdx. Finding: DI-2026-09-05-WEEKLY, reconciled plan item 10. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SL5chNkWnr4Gy6uuB8PeKS
hmishra2250
force-pushed
the
docs/capabilities-inbound-links
branch
from
September 7, 2026 11:57
4911819 to
4b56579
Compare
hmishra2250
changed the base branch from
docs/capabilities-and-licensing
to
main
September 8, 2026 06:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds one inbound link to
/capabilitiesfromapi-reference/endpoint/scrape.mdxand one fromfeatures/crawl.mdx. Two files, two additions, no other change.Stack
This targets
docs/capabilities-and-licensing, the branch of #1364, because that PR is what puts the capabilities page in the navigation. Retarget tomainonce #1364 merges. The change itself does not depend on the matrix work in #1364 and can be reviewed on its own.Why
DI-2026-09-05-WEEKLY, reconciled plan item 10.
capabilities.mdxis written as the canonical answer to what Firecrawl can do, but nothing in the main reading flow links to it. A page with no inbound link from the pages readers land on first is a page neither readers nor crawlers reach, whatever it says.Scrape and crawl are the two entry points where the question "can Firecrawl do X" actually comes up, so those are the two links.
Changes
api-reference/endpoint/scrape.mdx: a short Capabilities section above the existing Interactions section, in the same shape as that section. It says what the capabilities page answers: what a scrape can do to a page, which capabilities run self-hosted, and which endpoints accept a keyless request.features/crawl.mdx: one sentence after the intro bullet list, before the playground call to action. It makes the point that matters at that spot, which is that every page a crawl reaches goes through the same scrape pipeline, then links out for the list.Verification
mintlify broken-linksandmintlify validateon Node 22.23.2. 208 broken-link entries across 95 files and 17 validate warnings, the same counts as the base commit93dc8d5e. Neither new link is reported, and neither edited file appears in the broken-link output.git diff --name-only 93dc8d5e...HEADreturnsapi-reference/endpoint/scrape.mdxandfeatures/crawl.mdxonly.Overlap check
PR #1365 also edits
features/crawl.mdx. Its hunks are an insertion after line 242 and a replacement at lines 278-281. This change is at line 40 of the intro. No overlap, so the two can merge in either order.🤖 Generated with Claude Code
https://claude.ai/code/session_01SL5chNkWnr4Gy6uuB8PeKS