diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..80ea12782 --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,204 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`:firecrawl` **1.9.2**) and the v2 OpenAPI spec. Function names and parameter keys are generated from the OpenAPI spec. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.9"} +``` + +## Authenticate + +```elixir +# config/runtime.exs or config.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# Or pass api_key per call +{:ok, res} = Firecrawl.search_and_scrape( + [query: "site:docs.firecrawl.dev webhook retries"], + api_key: "fc-your-api-key" +) +``` + +All functions accept an optional `opts` keyword list as the last argument with `:api_key` (override per request) and `:base_url` (default: `"https://api.firecrawl.dev/v2"`). A nil or empty key enables the keyless free tier. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape( + query: "site:docs.firecrawl.dev crawl webhooks", + sources: [:web, :news], + limit: 10, + tbs: "qdr:m", + location: "San Francisco,California,United States", + scrape_options: [ + formats: ["markdown"], + only_main_content: true + ] +) + +web_results = res.body["data"]["web"] +``` + +### Parameters + +- `query` — string (required). The search query. Use `site:example.com` to limit results. +- `sources` — list of atoms, strings, or maps. Values: `:web`, `:news`, `:images` (or string equivalents, or `%{type: "web" | "news" | "images"}`). +- `categories` — list of atoms, strings, or maps. Values: `:github`, `:research`, `:pdf` (or equivalents). +- `include_domains` — list of strings. Domains to include. +- `exclude_domains` — list of strings. Domains to exclude. +- `limit` — integer. Maximum number of results. +- `tbs` — string. Time-based filter (e.g. `qdr:d`, `qdr:w`, `sbd:1,qdr:m`). +- `location` — string. Location for localized results. +- `country` — string. ISO 3166-1 alpha-2 country code (e.g. `"US"`). +- `ignore_invalid_urls` — boolean. Drop URLs that cannot be scraped. +- `timeout` — integer. Request timeout in milliseconds. +- `highlights` — boolean. Generate query-relevant highlights. Defaults to true. +- `enterprise` — list of strings. Enterprise ZDR options: `["zdr"]` for end-to-end Zero Data Retention, `["anon"]` for anonymized ZDR. +- `scrape_options` — keyword list. Options for scraping each search result (see Scrape parameters). + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com/pricing", + formats: [ + "markdown", + "links", + %{type: "json", prompt: "Extract plan names and prices."} + ], + only_main_content: true, + wait_for: 1000 +) + +doc = res.body["data"] +IO.puts(doc["markdown"]) +``` + +### Parameters + +- `url` — string (required). The URL to scrape. +- `formats` — list of format strings or maps. Requested output formats. + - String formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"branding"`, `"audio"`, `"video"`. + - Map formats: + - `%{type: "json", prompt: ..., schema: ...}` — JSON extraction. + - `%{type: "question", question: "..."}` — question-answer output. + - `%{type: "highlights", query: "..."}` — relevant source-text output. + - `%{type: "screenshot", fullPage: ..., quality: ..., viewport: ...}` — screenshot with options. + - `%{type: "changeTracking", modes: [...], tag: ...}` — change tracking. + - `%{type: "attributes", selectors: [%{selector: ..., attribute: ...}]}` — attribute extraction. +- `headers` — map. Custom request headers. +- `include_tags` — list of strings. HTML tags to include. +- `exclude_tags` — list of strings. HTML tags to exclude. +- `only_main_content` — boolean. Strip nav, footer, and boilerplate. +- `timeout` — integer. Timeout in milliseconds. Min: 1000, default: 60000, max: 300000. +- `wait_for` — integer. Wait for page to render (milliseconds). +- `mobile` — boolean. Use mobile viewport. +- `parsers` — list of strings or maps. Values: `"pdf"` or `%{type: "pdf", mode: "fast" | "auto" | "ocr", maxPages: integer}`. +- `actions` — list of action maps. Pre-scrape browser actions. Types: `wait`, `screenshot`, `click` (with optional `all`), `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. +- `location` — keyword list with `country:` and `languages:`. +- `skip_tls_verification` — boolean. Skip TLS verification. +- `remove_base64_images` — boolean. Drop base64 images from markdown. +- `block_ads` — boolean. Ad and cookie popup blocking. +- `proxy` — atom or string. Values: `:basic`, `:enhanced`, `:auto`. +- `max_age` — integer. Use cached data up to a maximum age (milliseconds). +- `min_age` — integer. Accept cached data only if at least this old (milliseconds). +- `store_in_cache` — boolean. Cache the result. +- `lockdown` — boolean. Serve only previously cached results. +- `redact_pii` — boolean. Redact PII. +- `audit_metadata` — keyword list with `username:`. Metadata for SIEM logging. +- `profile` — keyword list with `name:` and optional `save_changes:`. +- `zero_data_retention` — boolean. Enable zero data retention. + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` + +### Example + +```elixir +# First scrape to get a job ID +{:ok, scrape_res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] +) +job_id = get_in(scrape_res.body, ["data", "metadata", "scrapeId"]) + +# Code-based interaction +{:ok, res} = Firecrawl.interact_with_scrape_browser_session( + job_id, + code: "console.log(await page.title());", + language: :node, + timeout: 60 +) + +# Stop the session when done +{:ok, _} = Firecrawl.stop_interactive_scrape_browser_session(job_id) +``` + +### Parameters + +- `job_id` — string (required, first argument). The scrape job ID. +- `code` — string (required). Code to execute in the browser session. +- `language` — atom or string. Values: `:python`, `:node`, `:bash`. Default: `"node"`. +- `timeout` — integer. Execution timeout in seconds. +- `origin` — string. Optional origin label for telemetry. + +### Stop session + +`Firecrawl.stop_interactive_scrape_browser_session(job_id, opts \\ [])` — ends the browser session via `DELETE /scrape/{jobId}/interact`. A bang variant `stop_interactive_scrape_browser_session!/2` is also available. + +## Notes + +- The Elixir SDK exposes code-based interactions only — there is no `prompt` parameter on `interact_with_scrape_browser_session` (unlike JS, Python, and Rust SDKs). +- The client is OpenAPI-shaped: function names and parameter keys are generated from the spec. +- Each public function has a bang (`!`) variant that raises on error instead of returning `{:error, _}`. +- Parameter keys use snake_case; they are auto-converted to camelCase for the JSON body. +- Atom values (e.g. `:web`, `:node`, `:auto`) are converted to strings automatically. +- There are no deprecated aliases in this SDK. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..b7b6f1be1 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,241 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`com.firecrawl:firecrawl-java` **1.15.0**) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.15.0 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.15.0") +``` + +Requires Java 11+. + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); + +// Or from environment: +// FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + +Builder options: `apiKey` (String, falls back to `FIRECRAWL_API_KEY` env or `firecrawl.apiKey` system property), `apiUrl` (String, default `"https://api.firecrawl.dev"`), `timeoutMs` (long, default 300000), `maxRetries` (int, default 3), `backoffFactor` (double, default 0.5), `asyncExecutor` (Executor), `httpClient` (OkHttpClient). A null or blank key enables the keyless free tier. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +- `client.search(query)` → `SearchData` +- `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.SearchData; +import java.util.List; +import java.util.Map; + +SearchOptions options = SearchOptions.builder() + .sources(List.of("web", "news")) + .limit(10) + .tbs("qdr:m") + .scrapeOptions( + ScrapeOptions.builder() + .formats(List.of("markdown")) + .onlyMainContent(true) + .build() + ) + .build(); + +SearchData results = client.search("site:docs.firecrawl.dev crawl webhooks", options); +List> web = results.getWeb(); +``` + +**Wrong turn to avoid:** `search()` returns `SearchData` with `getWeb()`, `getNews()`, and `getImages()` — do not treat it as a directly iterable list. + +### Parameters + +- `query` — String (required). The search query. Use `site:example.com` to limit results to a domain. +- `options.sources` — `List`. Values: `"web"`, `"news"`, `"images"` or `{type: "web" | "news" | "images"}` maps. +- `options.categories` — `List`. Values: `"github"`, `"research"`, `"pdf"`. +- `options.includeDomains` — `List`. Domains to include. +- `options.excludeDomains` — `List`. Domains to exclude. +- `options.limit` — Integer. Maximum number of results. +- `options.tbs` — String. Time-based filter (e.g. `qdr:d`, `qdr:w`, `sbd:1,qdr:m`). +- `options.location` — String. Location for localized results. +- `options.ignoreInvalidURLs` — Boolean. Drop URLs that cannot be scraped. +- `options.timeout` — Integer. Request timeout in milliseconds. +- `options.highlights` — Boolean. Generate query-relevant highlights. Defaults to true. +- `options.scrapeOptions` — `ScrapeOptions`. Options for scraping each search result (see Scrape parameters). +- `options.integration` — String. Integration identifier. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +- `client.scrape(url)` → `Document` +- `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.JsonFormat; +import com.firecrawl.models.Document; +import java.util.List; + +ScrapeOptions options = ScrapeOptions.builder() + .formats(List.of( + "markdown", + "links", + JsonFormat.builder().prompt("Extract plan names and prices.").build() + )) + .onlyMainContent(true) + .waitFor(1000) + .build(); + +Document doc = client.scrape("https://example.com/pricing", options); +System.out.println(doc.getMarkdown()); +System.out.println(doc.getJson()); +``` + +### Parameters + +- `url` — String (required). The URL to scrape. +- `options.formats` — `List`. Format strings or format objects. + - String formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. + - Object formats: `JsonFormat.builder().prompt(...).schema(...).build()`, `QuestionFormat`, `HighlightsFormat`, or `Map` with `type` and options. +- `options.headers` — `Map`. Custom request headers. +- `options.includeTags` — `List`. HTML tags to include. +- `options.excludeTags` — `List`. HTML tags to exclude. +- `options.onlyMainContent` — Boolean. Strip nav, footer, and boilerplate. +- `options.timeout` — Integer. Timeout in milliseconds. +- `options.waitFor` — Integer. Wait for page to render (milliseconds). +- `options.mobile` — Boolean. Use mobile viewport. +- `options.parsers` — `List`. Values: `"pdf"` or `PdfParser` with `maxPages`, `pages`, `blocks`, `pageMarkers`. +- `options.actions` — `List>`. Pre-scrape browser actions. Types: `wait`, `screenshot`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. +- `options.location` — `LocationConfig` with `country` and `languages`. +- `options.skipTlsVerification` — Boolean. Skip TLS verification. +- `options.removeBase64Images` — Boolean. Drop base64 images from markdown. +- `options.blockAds` — Boolean. Ad and cookie popup blocking. +- `options.proxy` — String. `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or custom URL. +- `options.maxAge` — Long. Use cached data up to a maximum age (milliseconds). +- `options.storeInCache` — Boolean. Cache the result. +- `options.lockdown` — Boolean. Serve only previously cached results. +- `options.redactPII` — Boolean. Redact PII. +- `options.auditMetadata` — `AuditMetadata` with `username`. Metadata for SIEM logging. +- `options.integration` — String. Integration identifier. + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +- `client.interact(jobId, code)` — uses default language `"node"` and default timeout +- `client.interact(jobId, code, language, timeout)` — `timeout` is seconds (1–300), or null for default +- `client.interact(jobId, code, language, timeout, origin)` — optional `origin` for attribution + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; +import com.firecrawl.models.Document; +import com.firecrawl.models.ScrapeOptions; +import java.util.List; +import java.util.Map; + +// First scrape to get a job ID +Document doc = client.scrape( + "https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build() +); +Map metadata = doc.getMetadata(); +String jobId = (String) metadata.get("scrapeId"); + +// Code-based interaction +BrowserExecuteResponse result = client.interact( + jobId, + "console.log(await page.title());", + "node", + 60 +); +System.out.println(result.getStdout()); + +// Stop the session when done +client.stopInteractiveBrowser(jobId); +``` + +### Parameters + +- `jobId` — String (required). The scrape job ID. +- `code` — String (required). Code to run in the browser session. +- `language` — String. `"python"`, `"node"`, or `"bash"`. Default: `"node"`. +- `timeout` — Integer. Execution timeout in seconds (1–300). Null uses the API default (30). +- `origin` — String. Optional origin label for attribution. + +### Stop session + +`client.stopInteractiveBrowser(jobId)` → `BrowserDeleteResponse` — ends the browser session. Response: `isSuccess()`, `getSessionDurationMs()`, `getCreditsBilled()`, `getError()`. + +### Async variants + +All methods have `*Async` variants returning `CompletableFuture`: `scrapeAsync`, `searchAsync`, `interactAsync`, `stopInteractiveBrowserAsync`. + +## Notes + +- The Java SDK exposes code-based interactions only — there is no `prompt` parameter on `interact` (unlike JS, Python, and Rust SDKs). +- Deprecated aliases: `scrapeExecute` → `interact`; `deleteScrapeBrowser` → `stopInteractiveBrowser`. +- All options classes use the Builder pattern (`ScrapeOptions.builder()...build()`). +- Method overloading is used instead of optional/keyword arguments. +- All parameter names use camelCase (standard Java convention). +- `ScrapeOptions` supports `toBuilder()` for copying and modifying existing options. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchData.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/Document.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..67aa4f982 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,210 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`@mendable/firecrawl-js` **4.35.0**) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +npm install firecrawl +``` + +## Authenticate + +```ts +import { Firecrawl } from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, + // apiUrl: "https://api.firecrawl.dev" // optional; falls back to FIRECRAWL_API_URL or cloud default +}); +``` + +Constructor options: `apiKey` (string, falls back to `FIRECRAWL_API_KEY` env), `apiUrl` (string), `timeoutMs` (number), `maxRetries` (number), `backoffFactor` (number). A null or missing key enables the keyless free tier. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. For multi-step interactive flows, prefer `interact` over scrape-time `actions`. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries", { + sources: ["web", "news"], + limit: 10, + tbs: "qdr:m", + scrapeOptions: { + formats: ["markdown"], + onlyMainContent: true + } +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Web results are in `result.web`, news in `result.news`, images in `result.images`. + +### Parameters + +- `query` — string (required). The search query. Use `site:example.com` to limit results to a domain. +- `options.sources` — array of `"web" | "news" | "images"` or `{ type: "web" | "news" | "images" }`. Controls which result sources are searched. +- `options.categories` — array of `"github" | "research" | "pdf" | "developer"` or `{ type: ... }`. Filters results by category. `"research"` is a domain filter over ordinary web search (~14 academic domains). +- `options.includeDomains` — string array. Domains to include. Cannot be used with `excludeDomains`. +- `options.excludeDomains` — string array. Domains to exclude. Cannot be used with `includeDomains`. +- `options.limit` — number. Maximum number of results. +- `options.tbs` — string. Time-based filter (e.g. `qdr:d`, `qdr:w`, `sbd:1,qdr:m`). +- `options.location` — string. Location for localized results. +- `options.ignoreInvalidURLs` — boolean. Drop URLs that cannot be scraped by other endpoints. +- `options.timeout` — number. Request timeout in milliseconds. +- `options.highlights` — boolean. Generate query-relevant highlights. Defaults to true. +- `options.scrapeOptions` — `ScrapeOptions`. Options for scraping each search result (see Scrape parameters). +- `options.enterprise` — array of `"default" | "anon" | "zdr"`. Enterprise search options. `"zdr"` for Zero Data Retention, `"anon"` for anonymized search. +- `options.threatProtection` — object. Enterprise per-request override of the team's threat protection policy. +- `options.integration` — string. Integration identifier for server-side tracking. +- `options.origin` — string. Origin identifier. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com/pricing", { + formats: [ + "markdown", + "links", + { type: "json", prompt: "Extract plan names and prices." } + ], + onlyMainContent: true, + waitFor: 1000 +}); + +console.log(doc.markdown); +console.log(doc.json); +``` + +### Parameters + +- `url` — string (required). The URL to scrape. +- `options.formats` — array of format strings or format objects. Requested output formats. + - Plain string formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. + - Object-only formats (require at least `type`): + - `{ type: "json", prompt?, schema? }` — JSON extraction. At least one of `prompt` or `schema` is required. SDK rejects bare `"json"` string. + - `{ type: "question", question }` — question-answer style extraction. + - `{ type: "highlights", query }` — relevant source-text extraction. + - `{ type: "screenshot", fullPage?, quality?, viewport? }` — screenshot with options. + - `{ type: "changeTracking", modes, schema?, prompt?, tag? }` — `modes` is required, values: `"git-diff"`, `"json"`. + - `{ type: "attributes", selectors: [{ selector, attribute }] }` — attribute extraction. +- `options.headers` — `Record`. Custom request headers. +- `options.includeTags` — string array. HTML tags to include. +- `options.excludeTags` — string array. HTML tags to exclude. +- `options.onlyMainContent` — boolean. Strip nav, footer, and boilerplate. +- `options.timeout` — number. Timeout in milliseconds. +- `options.waitFor` — number. Wait for page to render (milliseconds). +- `options.mobile` — boolean. Use mobile viewport. +- `options.parsers` — array of `"pdf"` or `{ type: "pdf", mode?: "fast" | "auto" | "ocr", maxPages?, pages?, blocks?, pageMarkers? }`. +- `options.actions` — array of action objects. Pre-scrape browser actions. + - Action types: `wait` (`milliseconds` or `selector`), `screenshot`, `click` (`selector`), `write` (`text`), `press` (`key`), `scroll` (`direction: "up" | "down"`), `scrape`, `executeJavascript` (`script`), `pdf` (`format`, `landscape`, `scale`). +- `options.location` — `{ country?: string, languages?: string[] }`. Geo or language-aware scraping. +- `options.skipTlsVerification` — boolean. Skip TLS verification. +- `options.removeBase64Images` — boolean. Drop base64 images from markdown output. +- `options.fastMode` — boolean. Faster scrapes with reduced fidelity. +- `options.blockAds` — boolean. Ad and cookie popup blocking. +- `options.proxy` — `"basic" | "stealth" | "enhanced" | "auto"` or custom URL string. Proxy control. +- `options.maxAge` — number. Use cached data up to a maximum age (milliseconds). Set to `0` to bypass index reuse. +- `options.minAge` — number. Accept cached data only if at least this old (milliseconds). +- `options.storeInCache` — boolean. Cache the result. +- `options.lockdown` — boolean. Serve only previously cached results; never make outbound requests. +- `options.redactPII` — boolean or `{ mode, entities, replaceStyle }`. Redact personally identifiable information. +- `options.auditMetadata` — `{ username: string }`. Metadata for SIEM logging events. +- `options.profile` — `{ name: string, saveChanges?: boolean }`. Persistent browser profile across scrapes and interactions. +- `options.integration` — string. Integration identifier. +- `options.origin` — string. Origin identifier. + +## Interact + +### Why use it + +Use `interact` for code or natural-language control of the browser session tied to a scrape job (via `metadata.scrapeId`). The SDK requires at least one of `code` or `prompt`. For flows that go beyond quick pre-scrape tweaks, prefer `interact` over scrape-time `actions`. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { formats: ["markdown"] }); +const jobId = doc.metadata?.scrapeId; +if (!jobId) throw new Error("Missing scrapeId from scrape response"); + +// Natural-language interaction +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans." +}); + +// Code-based interaction +const codeResult = await client.interact(jobId, { + code: "console.log(await page.title());", + language: "node", + timeout: 60 +}); + +// Stop the session when done +await client.stopInteraction(jobId); +``` + +### Parameters + +- `jobId` — string (required). The scrape job ID from `document.metadata.scrapeId`. +- `args.code` — string. Code to run in the browser session (e.g. Playwright `page` usage). +- `args.prompt` — string. Natural-language instruction for the browser agent. +- At least one of `args.code` or `args.prompt` must be non-empty (SDK throws otherwise). +- `args.language` — `"python" | "node" | "bash"`. Runtime language. Default: `"node"`. +- `args.timeout` — number. Execution timeout in seconds. +- `args.origin` — string. Origin identifier. + +### Stop session + +`client.stopInteraction(jobId)` → `Promise` — ends the browser session. Response fields: `success`, `sessionDurationMs`, `creditsBilled`, `error`. + +## Notes + +- Deprecated client aliases: `scrapeUrl` → `scrape`; `scrapeExecute` → `interact`; `stopInteractiveBrowser` and `deleteScrapeBrowser` → `stopInteraction`. +- The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. +- Zod schemas passed to `formats` (for `json` or `changeTracking`) are converted to JSON Schema by the SDK. +- The package declares **Node.js >= 22** in `engines`. +- All parameters use camelCase. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/package.json` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..19a79ff37 --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,208 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl-py` **4.38.0**) and the v2 OpenAPI spec. Method names, parameters, and return types match the v2 client in `firecrawl/v2/client.py`. + +## Install + +```bash +pip install firecrawl-py +``` + +## Authenticate + +```py +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +# client = Firecrawl(api_key="fc-...", api_url="https://api.firecrawl.dev") +``` + +Constructor parameters: `api_key` (str, falls back to `FIRECRAWL_API_KEY` env), `api_url` (str, default `"https://api.firecrawl.dev"`), `timeout` (float, seconds), `max_retries` (int, default 3), `backoff_factor` (float, default 0.5). A null key enables the keyless free tier. An async variant `AsyncFirecrawl` is also available. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```py +results = client.search( + "site:docs.firecrawl.dev webhook retries", + sources=["web", "news"], + limit=10, + tbs="qdr:m", + scrape_options=ScrapeOptions( + formats=["markdown"], + only_main_content=True + ), +) + +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Web results are in `result.web`, news in `result.news`, images in `result.images`. + +### Parameters + +- `query` — str (required). The search query. Use `site:example.com` to limit results to a domain. +- `sources` — list of `"web" | "news" | "images"` or `Source` objects. Controls which result sources are searched. +- `categories` — list of `"github" | "research" | "pdf" | "developer"` or `Category` objects. Filters results by category. `"research"` is a domain filter over ordinary web search. +- `include_domains` — list of str. Domains to include. Cannot be used with `exclude_domains`. +- `exclude_domains` — list of str. Domains to exclude. Cannot be used with `include_domains`. +- `limit` — int. Maximum number of results. Default: 5. +- `tbs` — str. Time-based filter (e.g. `qdr:d`, `qdr:w`, `sbd:1,qdr:m`). +- `location` — str. Location for localized results. +- `ignore_invalid_urls` — bool. Drop URLs that cannot be scraped by other endpoints. +- `timeout` — int. Request timeout in milliseconds. Default: 300000. +- `highlights` — bool. Generate query-relevant highlights. Defaults to true. +- `scrape_options` — `ScrapeOptions`. Options for scraping each search result (see Scrape parameters). +- `enterprise` — list of str. Enterprise search options. `["zdr"]` for Zero Data Retention, `["anon"]` for anonymized search. +- `threat_protection` — `ThreatProtectionOptions`. Enterprise per-request override of threat protection policy. +- `integration` — str. Integration identifier. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```py +doc = client.scrape( + "https://example.com/pricing", + formats=[ + "markdown", + "links", + {"type": "json", "prompt": "Extract plan names and prices."}, + ], + only_main_content=True, + wait_for=1000, +) + +print(doc.markdown) +print(doc.json) +``` + +### Parameters + +- `url` — str (required). The URL to scrape. +- `formats` — list of format strings or dicts. Requested output formats. + - String formats: `"markdown"`, `"html"`, `"rawHtml"` (or `"raw_html"`), `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"` (or `"change_tracking"`), `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. + - Object-only formats: + - `{"type": "json", "prompt": ..., "schema": ...}` — JSON extraction. Use a dict, not the plain string `"json"`. + - `{"type": "question", "question": "..."}` — question-answer output. + - `{"type": "highlights", "query": "..."}` — relevant source-text output. + - `{"type": "screenshot", "full_page": ..., "quality": ..., "viewport": ...}` — screenshot with options. + - `{"type": "changeTracking", "modes": [...], "schema": ..., "prompt": ..., "tag": ...}` — `modes` is required. + - `{"type": "attributes", "selectors": [{"selector": ..., "attribute": ...}]}` — attribute extraction. +- `headers` — dict. Custom request headers. +- `include_tags` — list of str. HTML tags to include. +- `exclude_tags` — list of str. HTML tags to exclude. +- `only_main_content` — bool. Strip nav, footer, and boilerplate. +- `timeout` — int. Timeout in milliseconds. +- `wait_for` — int. Wait for page to render (milliseconds). +- `mobile` — bool. Use mobile viewport. +- `parsers` — list of `"pdf"` or `{"type": "pdf", "mode": "fast" | "auto" | "ocr", "max_pages": int}`. +- `actions` — list of action dicts. Pre-scrape browser actions. + - Action types: `wait` (`milliseconds` or `selector`), `screenshot`, `click` (`selector`), `write` (`text`), `press` (`key`), `scroll` (`direction: "up" | "down"`), `scrape`, `executeJavascript` (`script`), `pdf` (`format`, `landscape`, `scale`). +- `location` — dict with `country` and `languages`, or `Location` object. +- `skip_tls_verification` — bool. Skip TLS verification. +- `remove_base64_images` — bool. Drop base64 images from markdown output. +- `fast_mode` — bool. Faster scrapes with reduced fidelity. +- `block_ads` — bool. Ad and cookie popup blocking. +- `proxy` — str. `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. +- `max_age` — int. Use cached data up to a maximum age (milliseconds). +- `store_in_cache` — bool. Cache the result. +- `lockdown` — bool. Serve only previously cached results; never make outbound requests. +- `threat_protection` — `ThreatProtectionOptions`. Enterprise threat protection override. +- `audit_metadata` — `AuditMetadata` with `username: str`. Metadata for SIEM logging events. +- `profile` — dict with `name` and optional `saveChanges`. Persistent browser profile. +- `integration` — str. Integration identifier. + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` + +`prompt` is keyword-only. At least one of `code` or `prompt` must be non-empty. + +### Example + +```py +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.scrape_id if doc.metadata else None +if not job_id: + raise RuntimeError("Missing scrape_id from scrape response") + +# Natural-language interaction +result = client.interact(job_id, prompt="Click the pricing tab and summarize the plans.") + +# Code-based interaction +code_result = client.interact( + job_id, + code="print(await page.title())", + language="python", + timeout=60, +) + +# Stop the session when done +client.stop_interaction(job_id) +``` + +### Parameters + +- `job_id` — str (required). The scrape job ID from `document.metadata.scrape_id`. +- `code` — str. Code to run in the browser session (optional if `prompt` is set). +- `prompt` — str (keyword-only). Natural-language instruction for the browser agent (optional if `code` is set). +- `language` — `"python" | "node" | "bash"`. Runtime language. Default: `"node"`. +- `timeout` — int. Execution timeout in seconds (1–300). +- `origin` — str. Optional origin label. + +### Stop session + +`client.stop_interaction(job_id)` — ends the browser session. Returns `BrowserDeleteResponse` with `success`, `session_duration_ms`, `credits_billed`, `error`. + +## Notes + +- Deprecated aliases: `scrape_url` → `scrape`; `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. +- The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. +- `FirecrawlApp` is a backward-compatible alias for `Firecrawl`. +- All parameters use snake_case. The SDK normalizes to camelCase for the API. + +## Source Of Truth + +- `firecrawl/apps/python-sdk/pyproject.toml` +- `firecrawl/apps/python-sdk/firecrawl/__init__.py` +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..ff76cb4ce --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,239 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl` crate **2.16.0**) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +cargo add firecrawl +``` + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +`Client::new(api_key)` connects to the Firecrawl cloud. `Client::new_selfhosted(api_url, api_key)` connects to a self-hosted instance with an optional key. An empty or missing key enables the keyless free tier. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, SearchSource, ScrapeOptions, Format}; + +let options = SearchOptions { + sources: Some(vec![SearchSource::Web, SearchSource::News]), + limit: Some(10), + tbs: Some("qdr:m".to_string()), + scrape_options: Some(ScrapeOptions { + formats: Some(vec![Format::Markdown]), + only_main_content: Some(true), + ..Default::default() + }), + ..Default::default() +}; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", options) + .await?; + +if let Some(web) = &results.data.web { + for item in web { + // Each item is SearchResultOrDocument::WebResult or SearchResultOrDocument::Document + } +} +``` + +### Parameters + +- `query` — `impl AsRef` (required). The search query. Use `site:example.com` to limit results to a domain. +- `options.sources` — `Vec`. Values: `Web`, `News`, `Images`. +- `options.categories` — `Vec`. Values: `Github`, `Research`, `Pdf`. +- `options.include_domains` — `Vec`. Domains to include. Cannot be used with `exclude_domains`. +- `options.exclude_domains` — `Vec`. Domains to exclude. Cannot be used with `include_domains`. +- `options.limit` — `u32`. Maximum number of results. Default: 5, max: 20. +- `options.tbs` — `String`. Time-based filter (e.g. `qdr:d`, `qdr:w`, `sbd:1,qdr:m`). +- `options.location` — `String`. Location for localized results. +- `options.ignore_invalid_urls` — `bool`. Drop URLs that cannot be scraped. +- `options.timeout` — `u32`. Request timeout in milliseconds. +- `options.highlights` — `bool`. Generate query-relevant highlights. Defaults to true. +- `options.scrape_options` — `ScrapeOptions`. Options for scraping each search result (see Scrape parameters). +- `options.integration` — `String`. Integration identifier. +- `options.origin` — `String`. Origin label. Auto-set to `"rust-sdk@{version}"` if omitted. + +### Convenience method + +`client.search_and_scrape(query, limit)` → `Result, FirecrawlError>` — calls `search` with default `ScrapeOptions` and returns only the results that came back as full `Document` objects. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, JsonOptions}; + +let doc = client + .scrape("https://example.com/pricing", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links, Format::Json]), + json_options: Some(JsonOptions { + prompt: Some("Extract plan names and prices.".to_string()), + ..Default::default() + }), + only_main_content: Some(true), + wait_for: Some(1000), + ..Default::default() + }) + .await?; +``` + +### Parameters + +- `url` — `impl AsRef` (required). The URL to scrape. +- `options.formats` — `Vec`. Values: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`, `Question(QuestionFormat)`, `Highlights(HighlightsFormat)`. +- `options.headers` — `HashMap`. Custom request headers. +- `options.include_tags` — `Vec`. HTML tags to include. +- `options.exclude_tags` — `Vec`. HTML tags to exclude. +- `options.only_main_content` — `bool`. Strip nav, footer, and boilerplate. +- `options.timeout` — `u32`. Timeout in milliseconds. +- `options.wait_for` — `u32`. Wait for page to render (milliseconds). +- `options.mobile` — `bool`. Use mobile viewport. +- `options.parsers` — `Vec`. Values: `ParserConfig::Simple("pdf".to_string())` or `ParserConfig::Pdf { parser_type, mode, max_pages, pages, blocks, page_markers }`. +- `options.actions` — `Vec`. Pre-scrape browser actions. Types: `Wait`, `Screenshot`, `Click`, `Write`, `Press`, `Scroll`, `Scrape`, `ExecuteJavascript`, `Pdf`. +- `options.location` — `LocationConfig` with `country` and `languages`. +- `options.skip_tls_verification` — `bool`. Skip TLS verification. +- `options.remove_base64_images` — `bool`. Drop base64 images from markdown. +- `options.fast_mode` — `bool`. Faster scrapes with reduced fidelity. +- `options.block_ads` — `bool`. Ad and cookie popup blocking. +- `options.proxy` — `ProxyType`. Values: `Basic`, `Stealth`, `Enhanced`, `Auto`. +- `options.max_age` — `u32`. Use cached data up to a maximum age (seconds). +- `options.min_age` — `u32`. Accept cached data only if at least this old (seconds). +- `options.store_in_cache` — `bool`. Cache the result. +- `options.lockdown` — `bool`. Serve only previously cached results. +- `options.redact_pii` — `bool`. Redact PII (serialized as `redactPII`). +- `options.audit_metadata` — `AuditMetadata`. Metadata for SIEM logging. +- `options.profile` — `ProfileConfig` with `name` and `save_changes`. +- `options.integration` — `String`. Integration identifier. +- `options.json_options` — `JsonOptions` with `schema`, `system_prompt`, `prompt`. +- `options.screenshot_options` — `ScreenshotOptions` with `full_page`, `quality`, `viewport`. +- `options.change_tracking_options` — `ChangeTrackingOptions` with `modes` (`GitDiff`, `Json`), `schema`, `prompt`, `tag`. +- `options.attribute_selectors` — `Vec` with `selector`, `attribute`. + +### Convenience method + +`client.scrape_with_schema(url, schema, prompt)` → `Result` — scrapes with `Format::Json` and returns the extracted JSON. + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, ScrapeExecuteOptions, ScrapeExecuteLanguage}; + +// First scrape to get a job ID +let doc = client + .scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() + }) + .await?; + +// Use the scrapeId from metadata +let job_id = doc.metadata.as_ref() + .and_then(|m| m.get("scrapeId")) + .and_then(|v| v.as_str()) + .expect("Missing scrapeId"); + +// Natural-language interaction +let result = client + .interact(job_id, ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }) + .await?; + +// Code-based interaction +let code_result = client + .interact(job_id, ScrapeExecuteOptions { + code: Some("console.log(await page.title());".to_string()), + language: Some(ScrapeExecuteLanguage::Node), + timeout: Some(60), + ..Default::default() + }) + .await?; + +// Stop the session when done +let stopped = client.stop_interaction(job_id).await?; +``` + +### Parameters + +- `job_id` — `impl AsRef` (required). The scrape job ID. +- `options.code` — `String`. Code to run in the browser session. +- `options.prompt` — `String`. Natural-language instruction for the browser agent. +- At least one of `code` or `prompt` must be provided (SDK returns `FirecrawlError::Misuse` otherwise). +- `options.language` — `ScrapeExecuteLanguage`. Values: `Python`, `Node`, `Bash`. Default: `Node`. +- `options.timeout` — `u32`. Execution timeout in seconds. +- `options.origin` — `String`. Origin label. Auto-set to `"rust-sdk@{version}"` if omitted. + +### Stop session + +`client.stop_interaction(job_id)` → `Result` — ends the browser session. Response fields: `success`, `session_duration_ms`, `credits_billed`, `error`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. +- All option structs implement `Default`, enabling the `..Default::default()` pattern. +- Rust field names use snake_case; JSON serialization uses camelCase via `#[serde(rename_all = "camelCase")]`. +- Exception: `redact_pii` is serialized as `redactPII` (not `redactPii`). +- All public types are re-exported at the crate root: `use firecrawl::Client`. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl/apps/rust-sdk/src/lib.rs` +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl/apps/rust-sdk/src/types.rs` +- `firecrawl-docs/api-reference/v2-openapi.json`