> ## Documentation Index
> Fetch the complete documentation index at: https://firecrawl.web3doc.top/llms.txt
> Use this file to discover all available pages before exploring further.
# 抓取
> 将任何URL转换为干净数据
Firecrawl 将网页转换为 Markdown 格式，非常适合 LLM 应用。
* 它处理复杂性：代理、缓存、速率限制、被 JS 阻止的内容
* 处理动态内容：动态网站、JS 渲染的网站、PDF、图片
* 输出干净的 Markdown、结构化数据、截图或 HTML。
有关详细信息，请参阅[Scrape Endpoint API 参考](https://docs.firecrawl.dev/api-reference/endpoint/scrape)。
## 使用 Firecrawl 抓取 URL
### /scrape 端点
用于抓取 URL 并获取其内容。
### 安装
{' '}
```bash Python theme={null}
pip install firecrawl-py
```
{' '}
```bash Node theme={null}
npm install @mendable/firecrawl-js
```
{' '}
```bash Go theme={null}
go get github.com/mendableai/firecrawl-go
```
```yaml Rust theme={null}
# 将以下内容添加到您的Cargo.toml中
[dependencies]
firecrawl = "^1.0"
tokio = { version = "^1", features = ["full"] }
```
### 使用
{' '}
```python Python theme={null}
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key="fc-YOUR_API_KEY")
# 抓取一个网站:
scrape_result = app.scrape_url('firecrawl.dev', params={'formats': ['markdown', 'html']})
print(scrape_result)
```
{' '}
```js Node theme={null}
import FirecrawlApp, { ScrapeResponse } from '@mendable/firecrawl-js';
const app = new FirecrawlApp({apiKey: "fc-YOUR_API_KEY"});
// 抓取一个网站：
const scrapeResult = await app.scrapeUrl('firecrawl.dev', { formats: ['markdown', 'html'] }) as ScrapeResponse;
if (!scrapeResult.success) {
throw new Error(`抓取失败: ${scrapeResult.error}`)
}
console.log(scrapeResult)
```
{' '}
```go Go theme={null}
import (
"fmt"
"log"
"github.com/mendableai/firecrawl-go"
)
func main() {
// 使用你的API密钥初始化FirecrawlApp
apiKey := "fc-YOUR_API_KEY"
apiUrl := "https://api.firecrawl.dev"
version := "v1"
app, err := firecrawl.NewFirecrawlApp(apiKey, apiUrl, version)
if err != nil {
log.Fatalf("无法初始化FirecrawlApp: %v", err)
}
// 抓取一个网站
scrapeResult, err := app.ScrapeUrl("https://firecrawl.dev", map[string]any{
"formats": []string{"markdown", "html"},
})
if err != nil {
log.Fatalf("无法抓取URL: %v", err)
}
fmt.Println(scrapeResult)
}
```
{' '}
```rust Rust theme={null}
使用 firecrawl::{FirecrawlApp, scrape::{ScrapeOptions, ScrapeFormats}};
#[tokio::main]
async fn main() {
// 使用 API 密钥初始化 FirecrawlApp
let app = FirecrawlApp::new("fc-YOUR_API_KEY").expect("无法初始化 FirecrawlApp");
let options = ScrapeOptions {
formats: vec![ScrapeFormats::Markdown, ScrapeFormats::HTML].into(),
..Default::default()
};
let scrape_result = app.scrape_url("https://firecrawl.dev", options).await;
match scrape_result {
Ok(data) => println!("抓取结果:
{}", data.markdown.unwrap()),
Err(e) => eprintln!("地图失败: {}", e),
}
}
```
{' '}
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v1/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{
"url": "https://docs.firecrawl.dev",
"formats" : ["markdown", "html"]
}'
```
有关参数的更多详细信息，请参阅[API 参考](https://docs.firecrawl.dev/api-reference/endpoint/scrape)。
### 响应
SDK 将直接返回数据对象。cURL 将返回如下所示的有效负载。
```json theme={null}
{
"success": true,
"data" : {
"markdown": "Launch Week I来了！[查看我们的第二天发布🚀](https://www.firecrawl.dev/blog/launch-week-i-day-2-doubled-rate-limits)[💥免费获得两个月...",
"html": "
```python Python theme={null}
from firecrawl import FirecrawlApp
from pydantic import BaseModel, Field
# 使用你的API密钥初始化FirecrawlApp
app = FirecrawlApp(api_key='your_api_key')
class ExtractSchema(BaseModel):
company_mission: str
supports_sso: bool
is_open_source: bool
is_in_yc: bool
data = app.scrape_url('https://docs.firecrawl.dev/', {
'formats': ['json'],
'jsonOptions': {
'schema': ExtractSchema.model_json_schema(),
}
})
print(data["json"])
```
```js Node theme={null}
import FirecrawlApp from '@mendable/firecrawl-js';
import { z } from 'zod';
const app = new FirecrawlApp({
apiKey: 'fc-YOUR_API_KEY',
});
// 定义用于提取内容的架构
const schema = z.object({
company_mission: z.string(),
supports_sso: z.boolean(),
is_open_source: z.boolean(),
is_in_yc: z.boolean(),
});
const scrapeResult = await app.scrapeUrl('https://docs.firecrawl.dev/', {
formats: ['json'],
jsonOptions: { schema: schema },
});
if (!scrapeResult.success) {
throw new Error(`抓取失败: ${scrapeResult.error}`);
}
console.log(scrapeResult.extract);
```
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v1/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{
"url": "https://docs.firecrawl.dev/",
"formats": ["json"],
"jsonOptions": {
"schema": {
"type": "object",
"properties": {
"company_mission": {
"type": "string"
},
"supports_sso": {
"type": "boolean"
},
"is_open_source": {
"type": "boolean"
},
"is_in_yc": {
"type": "boolean"
}
},
"required": [
"company_mission",
"supports_sso",
"is_open_source",
"is_in_yc"
]
}
}
}'
```
输出：
```json JSON theme={null}
{
"success": true,
"data": {
"json": {
"company_mission": "训练一个安全的人工智能，利用您的技术资源回答客户和员工的问题，这样您的团队就不必这样做了",
"supports_sso": true,
"is_open_source": false,
"is_in_yc": true
},
"metadata": {
"title": "Mendable",
"description": "Mendable让您轻松构建AI聊天应用。摄取、定制，然后只需一行代码即可在您想要的任何地方部署。由SideGuide提供支持",
"robots": "follow, index",
"ogTitle": "Mendable",
"ogDescription": "Mendable让您轻松构建AI聊天应用。摄取、定制，然后只需一行代码即可在您想要的任何地方部署。由SideGuide提供支持",
"ogUrl": "https://docs.firecrawl.dev/",
"ogImage": "https://docs.firecrawl.dev/mendable_new_og1.png",
"ogLocaleAlternate": [],
"ogSiteName": "Mendable",
"sourceURL": "https://docs.firecrawl.dev/"
},
}
}
```
### 无模式提取（新功能）
现在可以通过仅传递`prompt`给端点来无模式提取。LLM 会选择数据的结构。
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v1/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{
"url": "https://docs.firecrawl.dev/",
"formats": ["json"],
"jsonOptions": {
"prompt": "从页面中提取公司使命。"
}
}'
```
输出：
```json JSON theme={null}
{
"success": true,
"data": {
"json": {
"company_mission": "训练一个安全的人工智能，使用您的技术资源回答客户和员工的问题，以便您的团队不必这样做",
},
"metadata": {
"title": "Mendable",
"description": "Mendable 允许您轻松构建 AI 聊天应用程序。摄取、自定义，然后只需一行代码即可在您想要的任何位置部署。由 SideGuide 提供支持",
"robots": "follow, index",
"ogTitle": "Mendable",
"ogDescription": "Mendable 允许您轻松构建 AI 聊天应用程序。摄取、自定义，然后只需一行代码即可在您想要的任何位置部署。由 SideGuide 提供支持",
"ogUrl": "https://docs.firecrawl.dev/",
"ogImage": "https://docs.firecrawl.dev/mendable_new_og1.png",
"ogLocaleAlternate": [],
"ogSiteName": "Mendable",
"sourceURL": "https://docs.firecrawl.dev/"
},
}
}
```
### 提取对象
`extract`对象接受以下参数：
* `schema`: 用于提取的模式。
* `systemPrompt`: 用于提取的系统提示。
* `prompt`: 用于无模式提取的提示。
## 与页面进行交互操作
Firecrawl 允许您在抓取内容之前对网页执行各种操作。这对于与动态内容互动、导航页面或访问需要用户互动的内容特别有用。
以下是如何使用操作导航到 google.com，搜索 Firecrawl，点击第一个结果，并截屏的示例。
重要的是几乎总是在其他操作前后使用`wait`操作，以留出足够的时间让页面加载。
### 示例
```python Python theme={null}
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key="fc-YOUR_API_KEY")
# 抓取一个网站:
scrape_result = app.scrape_url('firecrawl.dev',
params={
'formats': ['markdown', 'html'],
'actions': [
{"type": "wait", "milliseconds": 2000},
{"type": "click", "selector": "textarea[title=\"Search\"]"},
{"type": "wait", "milliseconds": 2000},
{"type": "write", "text": "firecrawl"},
{"type": "wait", "milliseconds": 2000},
{"type": "press", "key": "ENTER"},
{"type": "wait", "milliseconds": 3000},
{"type": "click", "selector": "h3"},
{"type": "wait", "milliseconds": 3000},
{"type": "scrape"},
{"type": "screenshot"}
]
}
)
print(scrape_result)
```
```js Node theme={null}
import FirecrawlApp, { ScrapeResponse } from '@mendable/firecrawl-js';
const app = new FirecrawlApp({apiKey: "fc-YOUR_API_KEY"});
// 抓取一个网站：
const scrapeResult = await app.scrapeUrl('firecrawl.dev', { formats: ['markdown', 'html'], actions: [
{ type: "wait", milliseconds: 2000 },
{ type: "click", selector: "textarea[title=\"Search\"]" },
{ type: "wait", milliseconds: 2000 },
{ type: "write", text: "firecrawl" },
{ type: "wait", milliseconds: 2000 },
{ type: "press", key: "ENTER" },
{ type: "wait", milliseconds: 3000 },
{ type: "click", selector: "h3" },
{ type: "scrape" },
{"type": "screenshot"}
] }) as ScrapeResponse;
if (!scrapeResult.success) {
throw new Error(`抓取失败: ${scrapeResult.error}`)
}
console.log(scrapeResult)
```
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v1/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{
"url": "google.com",
"formats": ["markdown"],
"actions": [
{"type": "wait", "milliseconds": 2000},
{"type": "click", "selector": "textarea[title=\"Search\"]"},
{"type": "wait", "milliseconds": 2000},
{"type": "write", "text": "firecrawl"},
{"type": "wait", "milliseconds": 2000},
{"type": "press", "key": "ENTER"},
{"type": "wait", "milliseconds": 3000},
{"type": "click", "selector": "h3"},
{"type": "wait", "milliseconds": 3000},
{"type": "screenshot"}
]
}'
```
### 输出
```json JSON theme={null}
{
"success": true,
"data": {
"markdown": "我们的首次Launch Week已经结束！[查看回顾 🚀](blog/firecrawl-launch-week-1-recap)...",
"actions": {
"screenshots": [
"https://alttmdsdujxrfnakrkyi.supabase.co/storage/v1/object/public/media/screenshot-75ef2d87-31e0-4349-a478-fb432a29e241.png"
],
"scrapes": [
{
"url": "https://www.firecrawl.dev/",
"html": "Firecrawl
"
}
]
},
"metadata": {
"title": "Home - Firecrawl",
"description": "Firecrawl爬取并将任何网站转换为干净的Markdown。",
"language": "en",
"keywords": "Firecrawl, Markdown, Data, Mendable, Langchain",
"robots": "follow, index",
"ogTitle": "Firecrawl",
"ogDescription": "将任何网站转换为LLM准备好的数据。",
"ogUrl": "https://www.firecrawl.dev/",
"ogImage": "https://www.firecrawl.dev/og.png?123",
"ogLocaleAlternate": [],
"ogSiteName": "Firecrawl",
"sourceURL": "http://google.com",
"statusCode": 200
}
}
}
```
有关操作参数的更多详细信息，请参阅[API参考](https://docs.firecrawl.dev/api-reference/endpoint/scrape)。
## 位置和语言
指定国家和首选语言以根据目标位置和语言偏好获取相关内容。
### 工作原理
当您指定位置设置时，Firecrawl 将使用适当的代理（如果有的话），并模拟相应的语言和时区设置。默认情况下，如果未指定位置，则设置为'US'。
### 使用
要使用位置和语言设置，请在请求体中包含`location`对象及其属性：
* `country`: ISO 3166-1 alpha-2 国家代码（例如，'US', 'AU', 'DE', 'JP'）。默认为'US'。
* `languages`: 请求的首选语言和地区优先级数组。默认为指定位置的语言。
```python Python theme={null}
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key="fc-YOUR_API_KEY")
# 抓取一个网站:
scrape_result = app.scrape_url('airbnb.com',
params={
'formats': ['markdown', 'html'],
'location': {
'country': 'BR',
'languages': ['pt-BR']
}
}
)
print(scrape_result)
```
```js Node theme={null}
import FirecrawlApp, { ScrapeResponse } from '@mendable/firecrawl-js';
const app = new FirecrawlApp({apiKey: "fc-YOUR_API_KEY"});
// 抓取一个网站：
const scrapeResult = await app.scrapeUrl('airbnb.com', { formats: ['markdown', 'html'], location: {
country: "BR",
languages: ["pt-BR"]
} }) as ScrapeResponse;
if (!scrapeResult.success) {
throw new Error(`抓取失败: ${scrapeResult.error}`)
}
console.log(scrapeResult)
```
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v1/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{
"url": "airbnb.com",
"formats": ["markdown"],
"location": {
"country": "BR",
"languages": ["pt-BR"]
}
}'
```
## 批量抓取多个 URL
现在可以同时批量抓取多个 URL。它接收起始 URL 和可选参数作为参数。params 参数允许您指定批量抓取作业的其他选项，例如输出格式。
### 工作原理
这与`/crawl`端点的工作原理非常相似。它提交一个批量抓取作业并返回一个作业 ID，用于检查批量抓取的状态。
SDK 提供同步和异步两种方法。同步方法将返回批量抓取作业的结果，而异步方法将返回一个作业 ID，您可以使用该 ID 来检查批量抓取的状态。
### 使用
```python Python theme={null}
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key="fc-YOUR_API_KEY")
# 抓取多个网站：
batch_scrape_result = app.batch_scrape_urls(['firecrawl.dev', 'mendable.ai'], {'formats': ['markdown', 'html']})
print(batch_scrape_result)
# 或者，您可以使用异步方法：
batch_scrape_job = app.async_batch_scrape_urls(['firecrawl.dev', 'mendable.ai'], {'formats': ['markdown', 'html']})
print(batch_scrape_job)
# (异步) 然后可以使用作业ID来检查批量抓取的状态：
batch_scrape_status = app.check_batch_scrape_status(batch_scrape_job['id'])
print(batch_scrape_status)
```
```js Node theme={null}
import FirecrawlApp, { ScrapeResponse } from '@mendable/firecrawl-js';
const app = new FirecrawlApp({apiKey: "fc-YOUR_API_KEY"});
// 抓取多个网站（同步）:
const batchScrapeResult = await app.batchScrapeUrls(['firecrawl.dev', 'mendable.ai'], { formats: ['markdown', 'html'] });
if (!batchScrapeResult.success) {
throw new Error(`抓取失败: ${batchScrapeResult.error}`);
}
// 输出批量抓取的所有结果:
console.log(batchScrapeResult);
// 或者，您可以使用异步方法:
const batchScrapeJob = await app.asyncBulkScrapeUrls(['firecrawl.dev', 'mendable.ai'], { formats: ['markdown', 'html'] });
console.log(batchScrapeJob);
// (异步) 然后可以使用任务ID来检查批量抓取的状态:
const batchScrapeStatus = await app.checkBatchScrapeStatus(batchScrapeJob.id);
console.log(batchScrapeStatus);
```
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v1/batch/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{
"urls": ["https://docs.firecrawl.dev", "https://docs.firecrawl.dev/sdks/overview"],
"formats" : ["markdown", "html"]
}'
```
### 响应
如果您使用的是 SDK 中的同步方法，它将返回批量抓取作业的结果。否则，它将返回一个作业 ID，您可以使用该 ID 来检查批量抓取的状态。
#### 同步方式
```json Completed theme={null}
{
"status": "completed",
"total": 36,
"completed": 36,
"creditsUsed": 36,
"expiresAt": "2024-00-00T00:00:00.000Z",
"next": "https://api.firecrawl.dev/v1/batch/scrape/123-456-789?skip=26",
"data": [
{
"markdown": "[Firecrawl Docs home page!...",
"html": "...",
"metadata": {
"title": "使用 Groq Llama 3 构建 '与网站聊天' | Firecrawl",
"language": "en",
"sourceURL": "https://docs.firecrawl.dev/learn/rag-llama3",
"description": "了解如何使用 Firecrawl、Groq Llama 3 和 Langchain 来构建一个 '与您的网站聊天' 机器人。",
"ogLocaleAlternate": [],
"statusCode": 200
}
},
...
]
}
```
#### 异步方式
然后，您可以通过调用 `/batch/scrape/{id}` 端点，使用作业 ID 检查批量抓取的状态。此端点应在作业仍在运行或作业完成后立即使用，**因为批量抓取作业会在 24 小时后过期**。
```json theme={null}
{
"success": true,
"id": "123-456-789",
"url": "https://api.firecrawl.dev/v1/batch/scrape/123-456-789"
}
```