快速入门 | Firecrawl

欢迎使用 Firecrawl

Firecrawl 是一个 API 服务，它接受一个 URL，抓取并转换为干净的 markdown。我们会抓取所有可访问的子页面，为您提供每个页面的干净 markdown。无需站点地图。

如何使用？

我们提供了易于使用的 API 和托管版本。您可以在这里找到游乐场和文档。如果您愿意，也可以自行托管后端。以下是开始使用的资源：

API: 文档
SDKs: Python, Node, Go, Rust
LLM 框架: Langchain (python), Langchain (js), Llama Index, Crew.ai, Composio, PraisonAI, Superinterface, Vectorize
低代码框架: Dify, Langflow, Flowise AI, Cargo, Pipedream
其他: Zapier, Pabbly Connect

需要 SDK 或集成？通过打开一个问题让我们知道。 自托管： 要进行自托管，请参考此处的指南。

API 密钥

要使用 API，您需要在Firecrawl上注册并获取一个 API 密钥。

功能

抓取: 抓取 URL 并以 LLM 准备就绪的格式（markdown、结构化数据通过LLM 提取、截图、html）获取其内容
爬虫: 抓取网页的所有 URL 并返回 LLM 准备就绪的格式的内容
地图: 输入一个网站并获取所有网站 URL - 非常快速
提取: 使用 AI 从单个页面、多个页面或整个网站获取结构化数据。

强大的功能

LLM 准备就绪的格式: markdown、结构化数据、截图、HTML、链接、元数据
硬核功能: 代理、反机器人机制、动态内容（js 渲染）、输出解析、编排
可定制性: 排除标签、使用自定义头部在认证墙后面爬取、最大爬取深度等…
媒体解析: pdfs、docx、图像。
可靠性第一: 设计用于获取您需要的数据——无论有多难。
操作: 点击、滚动、输入、等待等操作，然后再提取数据

您可以在我们的文档中找到 Firecrawl 的所有功能以及如何使用它们。

爬虫

用于抓取一个 URL 及其所有可访问的子页面。这将提交一个爬虫任务并返回一个作业 ID，以便检查爬虫的状态。

安装

pip install firecrawl-py

npm install @mendable/firecrawl-js

go get github.com/mendableai/firecrawl-go

# 将以下内容添加到您的Cargo.toml中
[dependencies]
firecrawl = "^1.0"
tokio = { version = "^1", features = ["full"] }

使用

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

# 爬取一个网站:
crawl_status = app.crawl_url(
  'https://firecrawl.dev',
  params={
    'limit': 100,
    'scrapeOptions': {'formats': ['markdown', 'html']}
  },
  poll_interval=30
)
print(crawl_status)

import FirecrawlApp from '@mendable/firecrawl-js';

const app = new FirecrawlApp({apiKey: "fc-YOUR_API_KEY"});

const crawlResponse = await app.crawlUrl('https://firecrawl.dev', {
  limit: 100,
  scrapeOptions: {
    formats: ['markdown', 'html'],
  }
})

if (!crawlResponse.success) {
  throw new Error(`Failed to crawl: ${crawlResponse.error}`)
}

console.log(crawlResponse)

import (
	"fmt"
	"log"

	"github.com/mendableai/firecrawl-go"
)

func main() {
	// 使用您的 API 密钥初始化 FirecrawlApp
	apiKey := "fc-YOUR_API_KEY"
	apiUrl := "https://api.firecrawl.dev"
	version := "v1"

	app, err := firecrawl.NewFirecrawlApp(apiKey, apiUrl, version)
	if err != nil {
		log.Fatalf("无法初始化 FirecrawlApp: %v", err)
	}

	// 爬取一个网站
	crawlStatus, err := app.CrawlUrl("https://firecrawl.dev", map[string]any{
		"limit": 100,
		"scrapeOptions": map[string]any{
			"formats": []string{"markdown", "html"},
		},
	})
	if err != nil {
		log.Fatalf("发送爬取请求失败: %v", err)
	}

	fmt.Println(crawlStatus) 
}

use firecrawl::{crawl::{CrawlOptions, CrawlScrapeOptions, CrawlScrapeFormats}, FirecrawlApp};

#[tokio::main]
async fn main() {
    // 使用API密钥初始化FirecrawlApp
    let app = FirecrawlApp::new("fc-YOUR_API_KEY").expect("无法初始化FirecrawlApp");

    // 爬取网站
    let crawl_options = CrawlOptions {
        scrape_options: CrawlScrapeOptions {
            formats: vec![CrawlScrapeFormats::Markdown, CrawlScrapeFormats::HTML].into(),
            ..Default::default()
        }.into(),
        limit: 100.into(),
        ..Default::default()
    };

    let crawl_result = app
        .crawl_url("https://mendable.ai", crawl_options)
        .await;

    match crawl_result {
        Ok(data) => println!("爬取结果（使用了{}积分）：
{:#?}", data.credits_used, data.data),
        Err(e) => eprintln!("爬取失败：{}", e),
    }
}

curl -X POST https://api.firecrawl.dev/v1/crawl \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "url": "https://docs.firecrawl.dev",
      "limit": 100,
      "scrapeOptions": {
        "formats": ["markdown", "html"]
      }
    }'

如果您使用的是 cURL 或 SDK 上的async crawl函数，这将返回一个ID，您可以使用该 ID 来检查爬虫的状态。

{
  "success": true,
  "id": "123-456-789",
  "url": "https://api.firecrawl.dev/v1/crawl/123-456-789"
}

检查爬虫任务

用于检查爬虫任务的状态并获取其结果。

crawl_status = app.check_crawl_status("<crawl_id>")
print(crawl_status)

Node;
const crawlResponse = await app.checkCrawlStatus('<crawl_id>');

if (!crawlResponse.success) {
  throw new Error(`Failed to check crawl status: ${crawlResponse.error}`);
}

console.log(crawlResponse);

// 获取爬取状态
crawlStatus, err := app.CheckCrawlStatus("<crawl_id>")

if err != nil {
  log.Fatalf("获取爬取状态失败: %v", err)
}

fmt.Println(crawlStatus)

let crawl_status = app.check_crawl_status(crawl_id).await;

match crawl_status {
    Ok(data) => println!("Crawl Status:\n{:#?}", data),
    Err(e) => eprintln!("Check crawl status failed: {}", e),
}

curl -X GET https://api.firecrawl.dev/v1/crawl/<crawl_id> \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY'

响应

根据爬虫的状态，响应将有所不同。对于未完成或超过 10MB 的大响应，会提供next URL 参数。您必须请求此 URL 以检索下一批 10MB 的数据。如果缺少next参数，则表示爬虫数据的结束。

{
  "status": "scraping",
  "total": 36,
  "completed": 10,
  "creditsUsed": 10,
  "expiresAt": "2024-00-00T00:00:00.000Z",
  "next": "https://api.firecrawl.dev/v1/crawl/123-456-789?skip=10",
  "data": [
    {
      "markdown": "[Firecrawl Docs home page![light logo](https://mintlify.s3-us-west-1.amazonaws.com/firecrawl/logo/light.svg)!...",
      "html": "<!DOCTYPE html><html lang=\"en\" class=\"js-focus-visible lg:[--scroll-mt:9.5rem]\" data-js-focus-visible=\"\">...",
      "metadata": {
        "title": "使用 Groq Llama 3 构建 '与网站聊天' | Firecrawl",
        "language": "en",
        "sourceURL": "https://docs.firecrawl.dev/learn/rag-llama3",
        "description": "学习如何使用 Firecrawl、Groq Llama 3 和 Langchain 构建一个 '与您的网站聊天' 机器人。",
        "ogLocaleAlternate": [],
        "statusCode": 200
      }
    },
    ...
  ]
}

{
  "status": "已完成",
  "total": 36,
  "completed": 36,
  "creditsUsed": 36,
  "expiresAt": "2024-00-00T00:00:00.000Z",
  "next": "https://api.firecrawl.dev/v1/crawl/123-456-789?skip=26",
  "data": [
    {
      "markdown": "[Firecrawl Docs home page![light logo](https://mintlify.s3-us-west-1.amazonaws.com/firecrawl/logo/light.svg)!...",
      "html": "<!DOCTYPE html><html lang=\"en\" class=\"js-focus-visible lg:[--scroll-mt:9.5rem]\" data-js-focus-visible=\"\">...",
      "metadata": {
        "title": "使用Groq Llama 3构建'与网站聊天' | Firecrawl",
        "language": "en",
        "sourceURL": "https://docs.firecrawl.dev/learn/rag-llama3",
        "description": "学习如何使用Firecrawl、Groq Llama 3和Langchain来构建一个'与您的网站聊天'机器人。",
        "ogLocaleAlternate": [],
        "statusCode": 200
      }
    },
    ...
  ]
}

抓取

要抓取单个 URL，请使用scrape_url方法。它接受 URL 作为参数并返回抓取的数据作为字典。

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

# 抓取一个网站:
scrape_result = app.scrape_url('firecrawl.dev', params={'formats': ['markdown', 'html']})
print(scrape_result)

import FirecrawlApp, { ScrapeResponse } from '@mendable/firecrawl-js';

const app = new FirecrawlApp({apiKey: "fc-YOUR_API_KEY"});

// 抓取一个网站：
const scrapeResult = await app.scrapeUrl('firecrawl.dev', { formats: ['markdown', 'html'] }) as ScrapeResponse;

if (!scrapeResult.success) {
  throw new Error(`抓取失败: ${scrapeResult.error}`)
}

console.log(scrapeResult)

import (
	"fmt"
	"log"

	"github.com/mendableai/firecrawl-go"
)

func main() {
	// 使用你的API密钥初始化FirecrawlApp
	apiKey := "fc-YOUR_API_KEY"
	apiUrl := "https://api.firecrawl.dev"
	version := "v1"

	app, err := firecrawl.NewFirecrawlApp(apiKey, apiUrl, version)
	if err != nil {
		log.Fatalf("无法初始化FirecrawlApp: %v", err)
	}

	// 抓取一个网站
	scrapeResult, err := app.ScrapeUrl("https://firecrawl.dev", map[string]any{
		"formats": []string{"markdown", "html"},
	})
	if err != nil {
		log.Fatalf("无法抓取URL: %v", err)
	}

	fmt.Println(scrapeResult)
}

使用 firecrawl::{FirecrawlApp, scrape::{ScrapeOptions, ScrapeFormats}};

#[tokio::main]
async fn main() {
    // 使用 API 密钥初始化 FirecrawlApp
    let app = FirecrawlApp::new("fc-YOUR_API_KEY").expect("无法初始化 FirecrawlApp");

    let options = ScrapeOptions {
        formats: vec![ScrapeFormats::Markdown, ScrapeFormats::HTML].into(),
        ..Default::default()
    };

    let scrape_result = app.scrape_url("https://firecrawl.dev", options).await;

    match scrape_result {
        Ok(data) => println!("抓取结果:
{}", data.markdown.unwrap()),
        Err(e) => eprintln!("地图失败: {}", e),
    }
}

curl -X POST https://api.firecrawl.dev/v1/scrape \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "url": "https://docs.firecrawl.dev",
      "formats" : ["markdown", "html"]
    }'

响应

SDK 将直接返回数据对象。cURL 将返回如下所示的有效载荷。

{
  "success": true,
  "data" : {
    "markdown": "Launch Week I来了！[查看我们的第二天发布🚀](https://www.firecrawl.dev/blog/launch-week-i-day-2-doubled-rate-limits)[💥免费获得两个月...",
    "html": "<!DOCTYPE html><html lang=\"zh\" class=\"light\" style=\"color-scheme: light;\"><body class=\"__variable_36bd41 __variable_d7dc5d font-inter ...",
    "metadata": {
      "title": "首页 - Firecrawl",
      "description": "Firecrawl抓取并将任何网站转换为干净的Markdown。",
      "language": "zh",
      "keywords": "Firecrawl,Markdown,Data,Mendable,Langchain",
      "robots": "follow, index",
      "ogTitle": "Firecrawl",
      "ogDescription": "将任何网站转换为LLM就绪数据。",
      "ogUrl": "https://www.firecrawl.dev/",
      "ogImage": "https://www.firecrawl.dev/og.png?123",
      "ogLocaleAlternate": [],
      "ogSiteName": "Firecrawl",
      "sourceURL": "https://firecrawl.dev",
      "statusCode": 200
    }
  }
}

提取

使用 LLM 提取，您可以轻松地从任何 URL 中提取结构化数据。我们支持 pydantic 模式，以便让您更容易使用。以下是如何使用它的方法： v1 目前仅在 node、python 和 cURL 上受支持。

from firecrawl import FirecrawlApp
from pydantic import BaseModel, Field

# 使用你的API密钥初始化FirecrawlApp
app = FirecrawlApp(api_key='your_api_key')

class ExtractSchema(BaseModel):
    company_mission: str
    supports_sso: bool
    is_open_source: bool
    is_in_yc: bool

data = app.scrape_url('https://docs.firecrawl.dev/', {
    'formats': ['json'],
    'jsonOptions': {
        'schema': ExtractSchema.model_json_schema(),
    }
})
print(data["json"])

import FirecrawlApp from '@mendable/firecrawl-js';
import { z } from 'zod';

const app = new FirecrawlApp({
  apiKey: 'fc-YOUR_API_KEY',
});

// 定义用于提取内容的架构
const schema = z.object({
  company_mission: z.string(),
  supports_sso: z.boolean(),
  is_open_source: z.boolean(),
  is_in_yc: z.boolean(),
});

const scrapeResult = await app.scrapeUrl('https://docs.firecrawl.dev/', {
  formats: ['json'],
  jsonOptions: { schema: schema },
});

if (!scrapeResult.success) {
  throw new Error(`抓取失败: ${scrapeResult.error}`);
}

console.log(scrapeResult.extract);

curl -X POST https://api.firecrawl.dev/v1/scrape \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "url": "https://docs.firecrawl.dev/",
      "formats": ["json"],
      "jsonOptions": {
        "schema": {
          "type": "object",
          "properties": {
            "company_mission": {
                      "type": "string"
            },
            "supports_sso": {
                      "type": "boolean"
            },
            "is_open_source": {
                      "type": "boolean"
            },
            "is_in_yc": {
                      "type": "boolean"
            }
          },
          "required": [
            "company_mission",
            "supports_sso",
            "is_open_source",
            "is_in_yc"
          ]
        }
      }
    }'

输出：

JSON

{
    "success": true,
    "data": {
      "json": {
        "company_mission": "训练一个安全的人工智能，利用您的技术资源回答客户和员工的问题，这样您的团队就不必这样做了",
        "supports_sso": true,
        "is_open_source": false,
        "is_in_yc": true
      },
      "metadata": {
        "title": "Mendable",
        "description": "Mendable让您轻松构建AI聊天应用。摄取、定制，然后只需一行代码即可在您想要的任何地方部署。由SideGuide提供支持",
        "robots": "follow, index",
        "ogTitle": "Mendable",
        "ogDescription": "Mendable让您轻松构建AI聊天应用。摄取、定制，然后只需一行代码即可在您想要的任何地方部署。由SideGuide提供支持",
        "ogUrl": "https://docs.firecrawl.dev/",
        "ogImage": "https://docs.firecrawl.dev/mendable_new_og1.png",
        "ogLocaleAlternate": [],
        "ogSiteName": "Mendable",
        "sourceURL": "https://docs.firecrawl.dev/"
      },
    }
}

无模式提取（新功能）

您现在可以通过仅传递一个prompt到端点来进行无模式提取。llm 选择数据的结构。

curl -X POST https://api.firecrawl.dev/v1/scrape \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "url": "https://docs.firecrawl.dev/",
      "formats": ["json"],
      "jsonOptions": {
        "prompt": "从页面中提取公司使命。"
      }
    }'

输出：

JSON

{
    "success": true,
    "data": {
      "json": {
        "company_mission": "训练一个安全的人工智能，使用您的技术资源回答客户和员工的问题，以便您的团队不必这样做",
      },
      "metadata": {
        "title": "Mendable",
        "description": "Mendable 允许您轻松构建 AI 聊天应用程序。摄取、自定义，然后只需一行代码即可在您想要的任何位置部署。由 SideGuide 提供支持",
        "robots": "follow, index",
        "ogTitle": "Mendable",
        "ogDescription": "Mendable 允许您轻松构建 AI 聊天应用程序。摄取、自定义，然后只需一行代码即可在您想要的任何位置部署。由 SideGuide 提供支持",
        "ogUrl": "https://docs.firecrawl.dev/",
        "ogImage": "https://docs.firecrawl.dev/mendable_new_og1.png",
        "ogLocaleAlternate": [],
        "ogSiteName": "Mendable",
        "sourceURL": "https://docs.firecrawl.dev/"
      },
    }
}

提取（v0）

app = FirecrawlApp(version="v0")

class ArticleSchema(BaseModel):
title: str
points: int
by: str
commentsURL: str

class TopArticlesSchema(BaseModel):
top: List[ArticleSchema] = Field(..., max_items=5, description="Top 5 stories")

data = app.scrape_url('https://news.ycombinator.com', {
'extractorOptions': {
'extractionSchema': TopArticlesSchema.model_json_schema(),
'mode': 'llm-extraction'
},
'pageOptions':{
'onlyMainContent': True
}
})
print(data["llm_extraction"])

import FirecrawlApp from "@mendable/firecrawl-js";
import { z } from "zod";

const app = new FirecrawlApp<"v0">({
  apiKey: "fc-YOUR_API_KEY",
  version: "v0",
});

const schema = z.object({
  top: z
    .array(
      z.object({
        title: z.string(),
        points: z.number(),
        by: z.string(),
        commentsURL: z.string(),
      })
    )
    .length(5)
    .describe("Hacker News上的前5个故事"),
});

const scrapeResult = await app.scrapeUrl("https://news.ycombinator.com", {
  extractorOptions: { extractionSchema: schema },
});

console.log(scrapeResult.data["llm_extraction"]);

import (
  "fmt"
  "log"

  "github.com/mendableai/firecrawl-go"
)

app, err := NewFirecrawlApp(TEST_API_KEY, API_URL)
if err != nil {
  log.Fatalf("Failed to initialize FirecrawlApp: %v", err)
}

params := map[string]any{
  "extractorOptions": ExtractorOptions{
    Mode:             "llm-extraction",
    ExtractionPrompt: "Based on the information on the page, find what the company's mission is and whether it supports SSO, and whether it is open source",
    ExtractionSchema: map[string]any{
      "type": "object",
      "properties": map[string]any{
        "company_mission": map[string]string{"type": "string"},
        "supports_sso":    map[string]string{"type": "boolean"},
        "is_open_source":  map[string]string{"type": "boolean"},
      },
      "required": []string{"company_mission", "supports_sso", "is_open_source"},
    },
  },
}

scrapeResult, err := app.ScrapeURL("https://news.ycombinator.com", params)
if err != nil {
  log.Fatalf("Failed to scrape URL: %v", err)
}
fmt.Println(scrapeResult.LLMExtraction)

use firecrawl::FirecrawlApp;

#[tokio::main]
async fn main() {
    let api_key = "YOUR_API_KEY";
    let api_url = "https://api.firecrawl.dev";
    let app = FirecrawlApp::new(api_key, api_url).expect("Failed to initialize FirecrawlApp");

    let json_schema = json!({
        "type": "object",
        "properties": {
            "top": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "title": {"type": "string"},
                        "points": {"type": "number"},
                        "by": {"type": "string"},
                        "commentsURL": {"type": "string"}
                    }
                }
            }
        }
    });
}

使用操作与页面交互

Firecrawl 允许您在抓取网页内容之前对网页执行各种操作。这对于与动态内容交互、浏览页面或访问需要用户互动的内容特别

​欢迎使用 Firecrawl

​如何使用？

​API 密钥

​功能

​强大的功能

​爬虫

​安装

​使用

​检查爬虫任务

​响应

​抓取

​响应

​提取

​无模式提取（新功能）

​提取（v0）

​使用操作与页面交互