Firecrawl

The unique identifier (UUID) of the crawl job to be canceled.

Error if any occurred during the execution of the action

Whether or not the action execution was successful or not

The base URL to start crawling from. This is the initial entry point for the web crawler.

Delay in milliseconds between requests to avoid overwhelming the server

Maximum number of pages to crawl. The crawl will stop once this limit is reached. Default is 10.

An optional webhook URL to receive real-time updates on the crawl job. Events include crawl start (`crawl.started`), page crawled (`crawl.page`), and crawl completion (`crawl.completed` or `crawl.failed`). The payload structure matches the `/scrape` endpoint response.

Maximum depth of subpages to crawl relative to the base URL. A depth of 0 crawls only the base URL, 1 crawls the base URL and its direct links, etc. Default is 2.

A list of Regular Expression (regex) patterns for URL paths to exclude from the crawl. URLs whose paths match any of these patterns will be ignored. For example, `"blog/archive/.*"` would exclude all paths under `/blog/archive/`.

A list of Regular Expression (regex) patterns for URL paths to include in the crawl. Only URLs whose paths match one of these patterns will be processed. For example, `"products/featured/.*"` would only include paths under `/products/featured/`.

If true, the crawler will ignore any sitemap.xml found on the website. Defaults to true.

Maximum depth for discovering new links, separate from crawling depth

If true, allows the crawler to navigate to pages that were linked from pages already visited (i.e., navigate 'backwards'). Defaults to false.

If true, allows the crawler to follow links that lead to external websites (different domains). Defaults to false.

Proxy configuration for requests

Maximum age in seconds for cached content. If content is older than this, it will be re-scraped

If true, emulate a mobile device when scraping

If true, ignore query parameters when determining if a URL has been visited

List of actions to perform on each page before scraping (e.g., clicking buttons, waiting)

Specifies the desired output formats for the scraped content from each page. Default is `["markdown"]`. If format is json, jsonOptions is required.

Custom HTTP headers to send with each request

Timeout in milliseconds for each page request. Default is 30000ms (30 seconds)

The duration in milliseconds to wait for page JavaScript to execute and content to load before scraping. Useful for pages with dynamically loaded content. Default is 123ms.

If true, block advertisements during scraping

Geolocation settings for the scraper

If true, attempt to parse PDF files encountered during crawling

A list of HTML tags to exclude from the scraped output. Content within these tags (and their children) will be removed before processing.

A list of HTML tags to specifically include in the scraped output. Only content within these tags will be processed. If empty or null, all relevant content is considered based on other options.

Options for JSON format extraction including schema and prompts

If true, store scraped content in cache for future use

If true, attempts to extract only the main content of each page, excluding common elements like headers, navigation bars, and footers. Default is true.

If true, remove base64-encoded images from the scraped content

If true, skip TLS certificate verification

Options for tracking changes between crawls

A dictionary containing the crawled data. This typically includes a job ID, status, and an array of page data if the crawl is complete and successful. The structure can vary based on the crawl outcome (e.g., success, failure, ongoing).

Error if any occurred during the execution of the action

Whether or not the action execution was successful or not

A list of URLs from which to extract data. Wildcards (e.g., `https://example.com/blog/*`) can be used for crawling multiple pages under a specific path.

Natural language query for information to extract from URL content. E.g., 'Extract the company mission, whether it supports SSO, etc.'

JSON object defining the desired structure for extracted data (e.g., field names, types). Dictates output format.

If `True`, allows crawling links outside initial domains in `urls`; if `False`, restricts to same domains.

A dictionary containing the structured data extracted from the URLs. The structure of this data will conform to the provided `schema` or the LLM's interpretation of the `prompt`.

Error if any occurred during the execution of the action

Whether or not the action execution was successful or not

Unique identifier (UUID) of the crawl job.

Details of the crawl job, including `status` (e.g., "scraping", "completed", "failed"), `total` pages attempted, `completed` successfully crawled pages, `creditsUsed`, and `expiresAt` (data expiration timestamp).

Error if any occurred during the execution of the action

Whether or not the action execution was successful or not

The primary base URL from which the mapping process will begin.

Maximum number of unique links/pages to discover and return; helps control mapping scope and duration.

Optional search query to guide URL mapping, prioritizing or finding specific page types. 'Smart' search is limited to 1000 initial results in Alpha, but overall mapping can exceed this.

If true, the crawler ignores sitemap.xml files, relying on page links for discovery.

If true, includes subdomains of the base URL in the mapping. E.g., if `url` is example.com, blog.example.com is mapped.

Dictionary containing the URL mapping results, typically a list of discovered URLs or a structured sitemap representation.

Error if any occurred during the execution of the action

Whether or not the action execution was successful or not

The fully qualified URL of the web page to scrape.

An optional list of browser actions (e.g., click, type, wait) to perform on the page *before* scraping occurs. Useful for interacting with dynamic content, filling forms, or navigating through page elements.

A list of desired output formats for the scraped content. Defaults to ['markdown']. If `json` is included, `jsonOptions` *must* be provided.

Maximum time in milliseconds to wait for the scraping request to complete. Defaults to 30000.

Time in milliseconds to wait for the page to load or for dynamic content to render before starting the scrape. Defaults to 0.

Location settings for the request

A list of HTML tags to specifically exclude from the output. Content within these tags will be removed.

A list of HTML tags to specifically include in the output. Content within these tags will be prioritized.

Options for JSON extraction

If true, attempts to extract only the main article content, excluding headers, footers, navigation bars, and ads. Defaults to true.

A dictionary containing the scraped data. Keys correspond to the requested formats (e.g., 'markdown', 'html', 'json', 'screenshot'), and values are the extracted content or metadata for those formats.

Error if any occurred during the execution of the action

Whether or not the action execution was successful or not

Language code for search results (e.g., 'en' for English, default 'en').

Maximum number of search results to return (1-10, default 5).

The search query to execute.

Country code to tailor search results (e.g., 'us' for United States, default 'us').

Desired output formats for scraped content of each search result (e.g., 'markdown', 'html'). If None, default scraping applies. Available: 'markdown', 'html', 'rawHtml', 'links', 'screenshot', 'screenshot@fullPage'.

Maximum time in milliseconds for search and scrape operations (1000-300000, default 60000).

List of search result items, each with details and potentially scraped content.

Error if any occurred during the execution of the action

Indicates if the overall search operation was successful.

Optional warning message about the search operation.

Whether or not the action execution was successful or not