抓取 | Firecrawl

使用Firecrawl进行抓取

Firecrawl将网页转换为Markdown，非常适合LLM应用程序。原因如下：

管理复杂性： 处理代理、缓存、速率限制和JavaScript阻止的内容，以实现顺畅的抓取。
动态内容： 从JavaScript渲染的网站、pdf、图片等中收集数据。
Markdown或结构化数据转换： 将收集到的数据转换为干净的Markdown或结构化输出，非常适合LLM处理或其他任务。

更多详情，请参阅抓取端点API参考。

抓取一个URL

/scrape端点

用于抓取URL并获取其内容。

安装

pip install firecrawl-py

npm install @mendable/firecrawl-js

go get github.com/mendableai/firecrawl-go

# Add this to your Cargo.toml
[dependencies]
firecrawl = "^0.1"
tokio = { version = "^1", features = ["full"] }
serde = { version = "^1.0", features = ["derive"] }
serde_json = "^1.0"
uuid = { version = "^1.10", features = ["v4"] }

[build-dependencies]
tokio = { version = "1", features = ["full"] }

使用示例

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="YOUR_API_KEY")

content = app.scrape_url("https://mendable.ai")

import { FirecrawlApp } from 'firecrawl-js';

const app = new FirecrawlApp({ apiKey: 'YOUR_API_KEY' });

const content = await app.scrapeUrl('https://mendable.ai');

import (
  "fmt"
  "log"

  "github.com/mendableai/firecrawl-go"
)

func main() {
  app, err := firecrawl.NewFirecrawlApp("YOUR_API_KEY")
  if err != nil {
    log.Fatalf("Failed to initialize FirecrawlApp: %v", err)
  }

  content, err := app.ScrapeURL("mendable.ai", nil)
  if err != nil {
    log.Fatalf("Failed to scrape URL: %v", err)
  }
  fmt.Println(content)
}

use firecrawl::FirecrawlApp;

#[tokio::main]
async fn main() {
    // 用API密钥初始化FirecrawlApp
    let api_key = "YOUR_API_KEY";
    let api_url = "https://api.firecrawl.dev";
    let app = FirecrawlApp::new(api_key, api_url).expect("Failed to initialize FirecrawlApp");

    // 抓取URL
    let scrape_result = app.scrape_url("https://mendable.ai", None).await;

    // 打印抓取结果
  match scrape_result {
    Ok(data) => println!("Scrape Result:
{}", data["markdown"]),
    Err(e) => eprintln!("Scrape failed: {}", e),
  }
}

curl -X POST https://api.firecrawl.dev/v0/scrape \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "url": "https://mendable.ai"
    }'

响应

SDK将直接返回数据对象。cURL将返回如下所示的负载。

{
  "success": true,
  "data": {
    "content": "原始内容 ",
    "markdown": "# Markdown内容",
    "provider": "web-scraper",
    "metadata": {
      "title": "Mendable | AI for CX and Sales",
      "description": "AI for CX and Sales",
      "language": null,
      "sourceURL": "https://www.mendable.ai/"
    }
  }
}