> ## Documentation Index
> Fetch the complete documentation index at: https://firecrawl.web3doc.top/llms.txt
> Use this file to discover all available pages before exploring further.

# 快速入门

> Firecrawl 允许您将整个网站转换为 LLM 就绪的 markdown

<img className="block" src="https://mintcdn.com/acme-9a1b4f3d/eo2-11L72PlIV-BL/images/hero.png?fit=max&auto=format&n=eo2-11L72PlIV-BL&q=85&s=21067f13b60a30ae7a6204728f042c66" alt="Hero Light" width="1280" height="704" data-path="images/hero.png" />

## 欢迎使用 Firecrawl

[Firecrawl](https://firecrawl.dev?ref=github) 是一个 API 服务，它接受一个 URL，抓取并转换为干净的 markdown。我们会抓取所有可访问的子页面，为您提供每个页面的干净 markdown。无需站点地图。

## 如何使用？

我们提供了易于使用的 API 和托管版本。您可以在 [这里](https://firecrawl.dev/playground) 找到游乐场和文档。如果您愿意，也可以自行托管后端。

以下是开始使用的资源：

* [x] **API**: [文档](https://docs.firecrawl.dev/api-reference/introduction)
* [x] **SDKs**: [Python](https://docs.firecrawl.dev/sdks/python), [Node](https://docs.firecrawl.dev/sdks/node), [Go](https://docs.firecrawl.dev/sdks/go), [Rust](https://docs.firecrawl.dev/sdks/rust)
* [x] **LLM 框架**: [Langchain (python)](https://python.langchain.com/docs/integrations/document_loaders/firecrawl/), [Langchain (js)](https://js.langchain.com/docs/integrations/document_loaders/web_loaders/firecrawl), [Llama Index](https://docs.llamaindex.ai/en/latest/examples/data_connectors/WebPageDemo/#using-firecrawl-reader), [Crew.ai](https://docs.crewai.com/), [Composio](https://composio.dev/tools/firecrawl/all), [PraisonAI](https://docs.praison.ai/firecrawl/), [Superinterface](https://superinterface.ai/docs/assistants/functions/firecrawl), [Vectorize](https://docs.vectorize.io/integrations/source-connectors/firecrawl)
* [x] **低代码框架**: [Dify](https://dify.ai/blog/dify-ai-blog-integrated-with-firecrawl), [Langflow](https://docs.langflow.org/), [Flowise AI](https://docs.flowiseai.com/integrations/langchain/document-loaders/firecrawl), [Cargo](https://docs.getcargo.io/integration/firecrawl), [Pipedream](https://pipedream.com/apps/firecrawl/)
* [x] **其他**: [Zapier](https://zapier.com/apps/firecrawl/integrations), [Pabbly Connect](https://www.pabbly.com/connect/integrations/firecrawl/)

需要 SDK 或集成？通过打开一个问题让我们知道。

**自托管：** 要进行自托管，请参考[此处](/contributing/self-host)的指南。

### API 密钥

要使用 API，您需要在[Firecrawl](https://firecrawl.dev)上注册并获取一个 API 密钥。

### 功能

* [**抓取**](#scraping): 抓取 URL 并以 LLM 准备就绪的格式（markdown、结构化数据通过[LLM 提取](#extraction)、截图、html）获取其内容
* [**爬虫**](#crawling): 抓取网页的所有 URL 并返回 LLM 准备就绪的格式的内容
* [**地图**](/features/map): 输入一个网站并获取所有网站 URL - 非常快速
* [**提取**](/features/extract): 使用 AI 从单个页面、多个页面或整个网站获取结构化数据。

### 强大的功能

* **LLM 准备就绪的格式**: markdown、结构化数据、截图、HTML、链接、元数据
* **硬核功能**: 代理、反机器人机制、动态内容（js 渲染）、输出解析、编排
* **可定制性**: 排除标签、使用自定义头部在认证墙后面爬取、最大爬取深度等...
* **媒体解析**: pdfs、docx、图像。
* **可靠性第一**: 设计用于获取您需要的数据——无论有多难。
* **操作**: 点击、滚动、输入、等待等操作，然后再提取数据

您可以在我们的[文档](https://docs.firecrawl.dev)中找到 Firecrawl 的所有功能以及如何使用它们。

## 爬虫

用于抓取一个 URL 及其所有可访问的子页面。这将提交一个爬虫任务并返回一个作业 ID，以便检查爬虫的状态。

### 安装

<CodeGroup>
  ```bash Python theme={null}
  pip install firecrawl-py
  ```

  ```bash Node theme={null}
  npm install @mendable/firecrawl-js
  ```

  ```bash Go theme={null}
  go get github.com/mendableai/firecrawl-go
  ```

  ```yaml Rust theme={null}
  # 将以下内容添加到您的Cargo.toml中
  [dependencies]
  firecrawl = "^1.0"
  tokio = { version = "^1", features = ["full"] }
  ```
</CodeGroup>

### 使用

<CodeGroup>
  ```python Python theme={null}
  from firecrawl import FirecrawlApp

  app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

  # 爬取一个网站:
  crawl_status = app.crawl_url(
    'https://firecrawl.dev',
    params={
      'limit': 100,
      'scrapeOptions': {'formats': ['markdown', 'html']}
    },
    poll_interval=30
  )
  print(crawl_status)
  ```

  ```js Node theme={null}
  import FirecrawlApp from '@mendable/firecrawl-js';

  const app = new FirecrawlApp({apiKey: "fc-YOUR_API_KEY"});

  const crawlResponse = await app.crawlUrl('https://firecrawl.dev', {
    limit: 100,
    scrapeOptions: {
      formats: ['markdown', 'html'],
    }
  })

  if (!crawlResponse.success) {
    throw new Error(`Failed to crawl: ${crawlResponse.error}`)
  }

  console.log(crawlResponse)
  ```

  ```go Go theme={null}
  import (
  	"fmt"
  	"log"

  	"github.com/mendableai/firecrawl-go"
  )

  func main() {
  	// 使用您的 API 密钥初始化 FirecrawlApp
  	apiKey := "fc-YOUR_API_KEY"
  	apiUrl := "https://api.firecrawl.dev"
  	version := "v1"

  	app, err := firecrawl.NewFirecrawlApp(apiKey, apiUrl, version)
  	if err != nil {
  		log.Fatalf("无法初始化 FirecrawlApp: %v", err)
  	}

  	// 爬取一个网站
  	crawlStatus, err := app.CrawlUrl("https://firecrawl.dev", map[string]any{
  		"limit": 100,
  		"scrapeOptions": map[string]any{
  			"formats": []string{"markdown", "html"},
  		},
  	})
  	if err != nil {
  		log.Fatalf("发送爬取请求失败: %v", err)
  	}

  	fmt.Println(crawlStatus) 
  }
  ```

  ```rust Rust theme={null}
  use firecrawl::{crawl::{CrawlOptions, CrawlScrapeOptions, CrawlScrapeFormats}, FirecrawlApp};

  #[tokio::main]
  async fn main() {
      // 使用API密钥初始化FirecrawlApp
      let app = FirecrawlApp::new("fc-YOUR_API_KEY").expect("无法初始化FirecrawlApp");

      // 爬取网站
      let crawl_options = CrawlOptions {
          scrape_options: CrawlScrapeOptions {
              formats: vec![CrawlScrapeFormats::Markdown, CrawlScrapeFormats::HTML].into(),
              ..Default::default()
          }.into(),
          limit: 100.into(),
          ..Default::default()
      };

      let crawl_result = app
          .crawl_url("https://mendable.ai", crawl_options)
          .await;

      match crawl_result {
          Ok(data) => println!("爬取结果（使用了{}积分）：
  {:#?}", data.credits_used, data.data),
          Err(e) => eprintln!("爬取失败：{}", e),
      }
  }
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.firecrawl.dev/v1/crawl \
      -H 'Content-Type: application/json' \
      -H 'Authorization: Bearer YOUR_API_KEY' \
      -d '{
        "url": "https://docs.firecrawl.dev",
        "limit": 100,
        "scrapeOptions": {
          "formats": ["markdown", "html"]
        }
      }'
  ```
</CodeGroup>

如果您使用的是 cURL 或 SDK 上的`async crawl`函数，这将返回一个`ID`，您可以使用该 ID 来检查爬虫的状态。

```json theme={null}
{
  "success": true,
  "id": "123-456-789",
  "url": "https://api.firecrawl.dev/v1/crawl/123-456-789"
}
```

### 检查爬虫任务

用于检查爬虫任务的状态并获取其结果。

<CodeGroup>
  ```python Python theme={null}
  crawl_status = app.check_crawl_status("<crawl_id>")
  print(crawl_status)
  ```

  ```js Node theme={null}
  Node;
  const crawlResponse = await app.checkCrawlStatus('<crawl_id>');

  if (!crawlResponse.success) {
    throw new Error(`Failed to check crawl status: ${crawlResponse.error}`);
  }

  console.log(crawlResponse);
  ```

  ```go Go theme={null}
  // 获取爬取状态
  crawlStatus, err := app.CheckCrawlStatus("<crawl_id>")

  if err != nil {
    log.Fatalf("获取爬取状态失败: %v", err)
  }

  fmt.Println(crawlStatus)
  ```

  ```rust Rust theme={null}
  let crawl_status = app.check_crawl_status(crawl_id).await;

  match crawl_status {
      Ok(data) => println!("Crawl Status:\n{:#?}", data),
      Err(e) => eprintln!("Check crawl status failed: {}", e),
  }
  ```

  ```bash cURL theme={null}
  curl -X GET https://api.firecrawl.dev/v1/crawl/<crawl_id> \
      -H 'Content-Type: application/json' \
      -H 'Authorization: Bearer YOUR_API_KEY'
  ```
</CodeGroup>

#### 响应

根据爬虫的状态，响应将有所不同。对于未完成或超过 10MB 的大响应，会提供`next` URL 参数。您必须请求此 URL 以检索下一批 10MB 的数据。如果缺少`next`参数，则表示爬虫数据的结束。

<CodeGroup>
  ```json theme={null}
  {
    "status": "scraping",
    "total": 36,
    "completed": 10,
    "creditsUsed": 10,
    "expiresAt": "2024-00-00T00:00:00.000Z",
    "next": "https://api.firecrawl.dev/v1/crawl/123-456-789?skip=10",
    "data": [
      {
        "markdown": "[Firecrawl Docs home page![light logo](https://mintlify.s3-us-west-1.amazonaws.com/firecrawl/logo/light.svg)!...",
        "html": "<!DOCTYPE html><html lang=\"en\" class=\"js-focus-visible lg:[--scroll-mt:9.5rem]\" data-js-focus-visible=\"\">...",
        "metadata": {
          "title": "使用 Groq Llama 3 构建 '与网站聊天' | Firecrawl",
          "language": "en",
          "sourceURL": "https://docs.firecrawl.dev/learn/rag-llama3",
          "description": "学习如何使用 Firecrawl、Groq Llama 3 和 Langchain 构建一个 '与您的网站聊天' 机器人。",
          "ogLocaleAlternate": [],
          "statusCode": 200
        }
      },
      ...
    ]
  }
  ```

  ```json theme={null}
  {
    "status": "已完成",
    "total": 36,
    "completed": 36,
    "creditsUsed": 36,
    "expiresAt": "2024-00-00T00:00:00.000Z",
    "next": "https://api.firecrawl.dev/v1/crawl/123-456-789?skip=26",
    "data": [
      {
        "markdown": "[Firecrawl Docs home page![light logo](https://mintlify.s3-us-west-1.amazonaws.com/firecrawl/logo/light.svg)!...",
        "html": "<!DOCTYPE html><html lang=\"en\" class=\"js-focus-visible lg:[--scroll-mt:9.5rem]\" data-js-focus-visible=\"\">...",
        "metadata": {
          "title": "使用Groq Llama 3构建'与网站聊天' | Firecrawl",
          "language": "en",
          "sourceURL": "https://docs.firecrawl.dev/learn/rag-llama3",
          "description": "学习如何使用Firecrawl、Groq Llama 3和Langchain来构建一个'与您的网站聊天'机器人。",
          "ogLocaleAlternate": [],
          "statusCode": 200
        }
      },
      ...
    ]
  }
  ```
</CodeGroup>

## 抓取

要抓取单个 URL，请使用`scrape_url`方法。它接受 URL 作为参数并返回抓取的数据作为字典。

<CodeGroup>
  ```python Python theme={null}
  from firecrawl import FirecrawlApp

  app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

  # 抓取一个网站:
  scrape_result = app.scrape_url('firecrawl.dev', params={'formats': ['markdown', 'html']})
  print(scrape_result)
  ```

  ```js Node theme={null}
  import FirecrawlApp, { ScrapeResponse } from '@mendable/firecrawl-js';

  const app = new FirecrawlApp({apiKey: "fc-YOUR_API_KEY"});

  // 抓取一个网站：
  const scrapeResult = await app.scrapeUrl('firecrawl.dev', { formats: ['markdown', 'html'] }) as ScrapeResponse;

  if (!scrapeResult.success) {
    throw new Error(`抓取失败: ${scrapeResult.error}`)
  }

  console.log(scrapeResult)
  ```

  ```go Go theme={null}
  import (
  	"fmt"
  	"log"

  	"github.com/mendableai/firecrawl-go"
  )

  func main() {
  	// 使用你的API密钥初始化FirecrawlApp
  	apiKey := "fc-YOUR_API_KEY"
  	apiUrl := "https://api.firecrawl.dev"
  	version := "v1"

  	app, err := firecrawl.NewFirecrawlApp(apiKey, apiUrl, version)
  	if err != nil {
  		log.Fatalf("无法初始化FirecrawlApp: %v", err)
  	}

  	// 抓取一个网站
  	scrapeResult, err := app.ScrapeUrl("https://firecrawl.dev", map[string]any{
  		"formats": []string{"markdown", "html"},
  	})
  	if err != nil {
  		log.Fatalf("无法抓取URL: %v", err)
  	}

  	fmt.Println(scrapeResult)
  }
  ```

  ```rust Rust theme={null}
  使用 firecrawl::{FirecrawlApp, scrape::{ScrapeOptions, ScrapeFormats}};

  #[tokio::main]
  async fn main() {
      // 使用 API 密钥初始化 FirecrawlApp
      let app = FirecrawlApp::new("fc-YOUR_API_KEY").expect("无法初始化 FirecrawlApp");

      let options = ScrapeOptions {
          formats: vec![ScrapeFormats::Markdown, ScrapeFormats::HTML].into(),
          ..Default::default()
      };

      let scrape_result = app.scrape_url("https://firecrawl.dev", options).await;

      match scrape_result {
          Ok(data) => println!("抓取结果:
  {}", data.markdown.unwrap()),
          Err(e) => eprintln!("地图失败: {}", e),
      }
  }
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.firecrawl.dev/v1/scrape \
      -H 'Content-Type: application/json' \
      -H 'Authorization: Bearer YOUR_API_KEY' \
      -d '{
        "url": "https://docs.firecrawl.dev",
        "formats" : ["markdown", "html"]
      }'
  ```
</CodeGroup>

### 响应

SDK 将直接返回数据对象。cURL 将返回如下所示的有效载荷。

```json theme={null}
{
  "success": true,
  "data" : {
    "markdown": "Launch Week I来了！[查看我们的第二天发布🚀](https://www.firecrawl.dev/blog/launch-week-i-day-2-doubled-rate-limits)[💥免费获得两个月...",
    "html": "<!DOCTYPE html><html lang=\"zh\" class=\"light\" style=\"color-scheme: light;\"><body class=\"__variable_36bd41 __variable_d7dc5d font-inter ...",
    "metadata": {
      "title": "首页 - Firecrawl",
      "description": "Firecrawl抓取并将任何网站转换为干净的Markdown。",
      "language": "zh",
      "keywords": "Firecrawl,Markdown,Data,Mendable,Langchain",
      "robots": "follow, index",
      "ogTitle": "Firecrawl",
      "ogDescription": "将任何网站转换为LLM就绪数据。",
      "ogUrl": "https://www.firecrawl.dev/",
      "ogImage": "https://www.firecrawl.dev/og.png?123",
      "ogLocaleAlternate": [],
      "ogSiteName": "Firecrawl",
      "sourceURL": "https://firecrawl.dev",
      "statusCode": 200
    }
  }
}
```

## 提取

使用 LLM 提取，您可以轻松地从任何 URL 中提取结构化数据。我们支持 pydantic 模式，以便让您更容易使用。以下是如何使用它的方法：

v1 目前仅在 node、python 和 cURL 上受支持。

<CodeGroup>
  ```python Python theme={null}
  from firecrawl import FirecrawlApp
  from pydantic import BaseModel, Field

  # 使用你的API密钥初始化FirecrawlApp
  app = FirecrawlApp(api_key='your_api_key')

  class ExtractSchema(BaseModel):
      company_mission: str
      supports_sso: bool
      is_open_source: bool
      is_in_yc: bool

  data = app.scrape_url('https://docs.firecrawl.dev/', {
      'formats': ['json'],
      'jsonOptions': {
          'schema': ExtractSchema.model_json_schema(),
      }
  })
  print(data["json"])
  ```

  ```js Node theme={null}
  import FirecrawlApp from '@mendable/firecrawl-js';
  import { z } from 'zod';

  const app = new FirecrawlApp({
    apiKey: 'fc-YOUR_API_KEY',
  });

  // 定义用于提取内容的架构
  const schema = z.object({
    company_mission: z.string(),
    supports_sso: z.boolean(),
    is_open_source: z.boolean(),
    is_in_yc: z.boolean(),
  });

  const scrapeResult = await app.scrapeUrl('https://docs.firecrawl.dev/', {
    formats: ['json'],
    jsonOptions: { schema: schema },
  });

  if (!scrapeResult.success) {
    throw new Error(`抓取失败: ${scrapeResult.error}`);
  }

  console.log(scrapeResult.extract);
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.firecrawl.dev/v1/scrape \
      -H 'Content-Type: application/json' \
      -H 'Authorization: Bearer YOUR_API_KEY' \
      -d '{
        "url": "https://docs.firecrawl.dev/",
        "formats": ["json"],
        "jsonOptions": {
          "schema": {
            "type": "object",
            "properties": {
              "company_mission": {
                        "type": "string"
              },
              "supports_sso": {
                        "type": "boolean"
              },
              "is_open_source": {
                        "type": "boolean"
              },
              "is_in_yc": {
                        "type": "boolean"
              }
            },
            "required": [
              "company_mission",
              "supports_sso",
              "is_open_source",
              "is_in_yc"
            ]
          }
        }
      }'
  ```
</CodeGroup>

输出：

```json JSON theme={null}
{
    "success": true,
    "data": {
      "json": {
        "company_mission": "训练一个安全的人工智能，利用您的技术资源回答客户和员工的问题，这样您的团队就不必这样做了",
        "supports_sso": true,
        "is_open_source": false,
        "is_in_yc": true
      },
      "metadata": {
        "title": "Mendable",
        "description": "Mendable让您轻松构建AI聊天应用。摄取、定制，然后只需一行代码即可在您想要的任何地方部署。由SideGuide提供支持",
        "robots": "follow, index",
        "ogTitle": "Mendable",
        "ogDescription": "Mendable让您轻松构建AI聊天应用。摄取、定制，然后只需一行代码即可在您想要的任何地方部署。由SideGuide提供支持",
        "ogUrl": "https://docs.firecrawl.dev/",
        "ogImage": "https://docs.firecrawl.dev/mendable_new_og1.png",
        "ogLocaleAlternate": [],
        "ogSiteName": "Mendable",
        "sourceURL": "https://docs.firecrawl.dev/"
      },
    }
}
```

### 无模式提取（新功能）

您现在可以通过仅传递一个`prompt`到端点来进行无模式提取。llm 选择数据的结构。

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST https://api.firecrawl.dev/v1/scrape \
      -H 'Content-Type: application/json' \
      -H 'Authorization: Bearer YOUR_API_KEY' \
      -d '{
        "url": "https://docs.firecrawl.dev/",
        "formats": ["json"],
        "jsonOptions": {
          "prompt": "从页面中提取公司使命。"
        }
      }'
  ```
</CodeGroup>

输出：

```json JSON theme={null}
{
    "success": true,
    "data": {
      "json": {
        "company_mission": "训练一个安全的人工智能，使用您的技术资源回答客户和员工的问题，以便您的团队不必这样做",
      },
      "metadata": {
        "title": "Mendable",
        "description": "Mendable 允许您轻松构建 AI 聊天应用程序。摄取、自定义，然后只需一行代码即可在您想要的任何位置部署。由 SideGuide 提供支持",
        "robots": "follow, index",
        "ogTitle": "Mendable",
        "ogDescription": "Mendable 允许您轻松构建 AI 聊天应用程序。摄取、自定义，然后只需一行代码即可在您想要的任何位置部署。由 SideGuide 提供支持",
        "ogUrl": "https://docs.firecrawl.dev/",
        "ogImage": "https://docs.firecrawl.dev/mendable_new_og1.png",
        "ogLocaleAlternate": [],
        "ogSiteName": "Mendable",
        "sourceURL": "https://docs.firecrawl.dev/"
      },
    }
}
```

### 提取（v0）

<CodeGroup>
  ```python Python theme={null}
  app = FirecrawlApp(version="v0")

  class ArticleSchema(BaseModel):
  title: str
  points: int
  by: str
  commentsURL: str

  class TopArticlesSchema(BaseModel):
  top: List[ArticleSchema] = Field(..., max_items=5, description="Top 5 stories")

  data = app.scrape_url('https://news.ycombinator.com', {
  'extractorOptions': {
  'extractionSchema': TopArticlesSchema.model_json_schema(),
  'mode': 'llm-extraction'
  },
  'pageOptions':{
  'onlyMainContent': True
  }
  })
  print(data["llm_extraction"])

  ```

  ```js JavaScript theme={null}
  import FirecrawlApp from "@mendable/firecrawl-js";
  import { z } from "zod";

  const app = new FirecrawlApp<"v0">({
    apiKey: "fc-YOUR_API_KEY",
    version: "v0",
  });

  const schema = z.object({
    top: z
      .array(
        z.object({
          title: z.string(),
          points: z.number(),
          by: z.string(),
          commentsURL: z.string(),
        })
      )
      .length(5)
      .describe("Hacker News上的前5个故事"),
  });

  const scrapeResult = await app.scrapeUrl("https://news.ycombinator.com", {
    extractorOptions: { extractionSchema: schema },
  });

  console.log(scrapeResult.data["llm_extraction"]);
  ```

  ```go Go theme={null}
  import (
    "fmt"
    "log"

    "github.com/mendableai/firecrawl-go"
  )

  app, err := NewFirecrawlApp(TEST_API_KEY, API_URL)
  if err != nil {
    log.Fatalf("Failed to initialize FirecrawlApp: %v", err)
  }

  params := map[string]any{
    "extractorOptions": ExtractorOptions{
      Mode:             "llm-extraction",
      ExtractionPrompt: "Based on the information on the page, find what the company's mission is and whether it supports SSO, and whether it is open source",
      ExtractionSchema: map[string]any{
        "type": "object",
        "properties": map[string]any{
          "company_mission": map[string]string{"type": "string"},
          "supports_sso":    map[string]string{"type": "boolean"},
          "is_open_source":  map[string]string{"type": "boolean"},
        },
        "required": []string{"company_mission", "supports_sso", "is_open_source"},
      },
    },
  }

  scrapeResult, err := app.ScrapeURL("https://news.ycombinator.com", params)
  if err != nil {
    log.Fatalf("Failed to scrape URL: %v", err)
  }
  fmt.Println(scrapeResult.LLMExtraction)
  ```

  ```rust Rust theme={null}
  use firecrawl::FirecrawlApp;

  #[tokio::main]
  async fn main() {
      let api_key = "YOUR_API_KEY";
      let api_url = "https://api.firecrawl.dev";
      let app = FirecrawlApp::new(api_key, api_url).expect("Failed to initialize FirecrawlApp");

      let json_schema = json!({
          "type": "object",
          "properties": {
              "top": {
                  "type": "array",
                  "items": {
                      "type": "object",
                      "properties": {
                          "title": {"type": "string"},
                          "points": {"type": "number"},
                          "by": {"type": "string"},
                          "commentsURL": {"type": "string"}
                      }
                  }
              }
          }
      });
  }
  ```
</CodeGroup>

## 使用操作与页面交互

Firecrawl 允许您在抓取网页内容之前对网页执行各种操作。这对于与动态内容交互、浏览页面或访问需要用户互动的内容特别