批量抓取多个URLs
你现在可以同时批量抓取多个URL。它接受起始URL和可选参数作为参数。params参数允许你指定批量抓取作业的额外选项，例如输出格式。工作原理
它与/crawl 端点的工作方式非常相似。它提交一个批量抓取作业并返回一个作业ID，用于检查批量抓取的状态。
SDK提供了两种方法，同步和异步。同步方法将返回批量抓取作业的结果，而异步方法将返回一个作业ID，你可以使用该ID来检查批量抓取的状态。
使用方法
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key="fc-YOUR_API_KEY")
# 抓取多个网站：
batch_scrape_result = app.batch_scrape_urls(['firecrawl.dev', 'mendable.ai'], {'formats': ['markdown', 'html']})
print(batch_scrape_result)
# 或者，您可以使用异步方法：
batch_scrape_job = app.async_batch_scrape_urls(['firecrawl.dev', 'mendable.ai'], {'formats': ['markdown', 'html']})
print(batch_scrape_job)
# (异步) 然后可以使用作业ID来检查批量抓取的状态：
batch_scrape_status = app.check_batch_scrape_status(batch_scrape_job['id'])
print(batch_scrape_status)
import FirecrawlApp, { ScrapeResponse } from '@mendable/firecrawl-js';
const app = new FirecrawlApp({apiKey: "fc-YOUR_API_KEY"});
// 抓取多个网站（同步）:
const batchScrapeResult = await app.batchScrapeUrls(['firecrawl.dev', 'mendable.ai'], { formats: ['markdown', 'html'] });
if (!batchScrapeResult.success) {
throw new Error(`抓取失败: ${batchScrapeResult.error}`);
}
// 输出批量抓取的所有结果:
console.log(batchScrapeResult);
// 或者，您可以使用异步方法:
const batchScrapeJob = await app.asyncBulkScrapeUrls(['firecrawl.dev', 'mendable.ai'], { formats: ['markdown', 'html'] });
console.log(batchScrapeJob);
// (异步) 然后可以使用任务ID来检查批量抓取的状态:
const batchScrapeStatus = await app.checkBatchScrapeStatus(batchScrapeJob.id);
console.log(batchScrapeStatus);
curl -X POST https://api.firecrawl.dev/v1/batch/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{
"urls": ["https://docs.firecrawl.dev", "https://docs.firecrawl.dev/sdks/overview"],
"formats" : ["markdown", "html"]
}'
响应
如果你使用的是SDK的同步方法，它将返回批量抓取作业的结果。否则，它将返回一个作业ID，你可以使用该ID来检查批量抓取的状态。同步
Completed
{
"status": "completed",
"total": 36,
"completed": 36,
"creditsUsed": 36,
"expiresAt": "2024-00-00T00:00:00.000Z",
"next": "https://api.firecrawl.dev/v1/batch/scrape/123-456-789?skip=26",
"data": [
{
"markdown": "[Firecrawl Docs home page!...",
"html": "<!DOCTYPE html><html lang=\"en\" class=\"js-focus-visible lg:[--scroll-mt:9.5rem]\" data-js-focus-visible=\"\">...",
"metadata": {
"title": "使用 Groq Llama 3 构建 '与网站聊天' | Firecrawl",
"language": "en",
"sourceURL": "https://docs.firecrawl.dev/learn/rag-llama3",
"description": "了解如何使用 Firecrawl、Groq Llama 3 和 Langchain 来构建一个 '与您的网站聊天' 机器人。",
"ogLocaleAlternate": [],
"statusCode": 200
}
},
...
]
}
异步
然后你可以使用作业ID通过调用/batch/scrape/{id} 端点来检查批量抓取的状态。这个端点应在作业仍在运行时或在作业刚刚完成后使用 因为批量抓取作业会在24小时后过期。
{
"success": true,
"id": "123-456-789",
"url": "https://api.firecrawl.dev/v1/batch/scrape/123-456-789"
}
带提取的批量抓取
你也可以使用批量抓取端点从页面中提取结构化数据。如果你想从一个URL列表中获得相同的结构化数据，这将非常有用。from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key="fc-YOUR_API_KEY")
# 批量抓取多个网站:
batch_scrape_result = app.batch_scrape_urls(
['https://docs.firecrawl.dev', 'https://docs.firecrawl.dev/sdks/overview'],
{
'formats': ['extract'],
'extract': {
'prompt': '从页面提取标题和描述。',
'schema': {
'type': 'object',
'properties': {
'title': {'type': 'string'},
'description': {'type': 'string'}
},
'required': ['title', 'description']
}
}
}
)
print(batch_scrape_result)
# 或者，你可以使用异步方法:
batch_scrape_job = app.async_batch_scrape_urls(
['https://docs.firecrawl.dev', 'https://docs.firecrawl.dev/sdks/overview'],
{
'formats': ['extract'],
'extract': {
'prompt': '从页面提取标题和描述。',
'schema': {
'type': 'object',
'properties': {
'title': {'type': 'string'},
'description': {'type': 'string'}
},
'required': ['title', 'description']
}
}
}
)
print(batch_scrape_job)
# (异步) 然后你可以使用任务ID来检查批量抓取的状态:
batch_scrape_status = app.check_batch_scrape_status(batch_scrape_job['id'])
print(batch_scrape_status)
import FirecrawlApp, { ScrapeResponse } from '@mendable/firecrawl-js';
const app = new FirecrawlApp({ apiKey: 'fc-YOUR_API_KEY' });
// 定义要提取内容的架构
const schema = {
type: 'object',
properties: {
title: { type: 'string' },
description: { type: 'string' },
},
required: ['title', 'description'],
};
// 同步抓取多个网站：
const batchScrapeResult = await app.batchScrapeUrls(['https://docs.firecrawl.dev', 'https://docs.firecrawl.dev/sdks/overview'], {
formats: ['extract'],
extract: {
prompt: '从页面中提取标题和描述。',
schema: schema,
},
});
if (!batchScrapeResult.success) {
throw new Error(`抓取失败: ${batchScrapeResult.error}`);
}
// 输出批量抓取的所有结果：
console.log(batchScrapeResult);
// 或者，你可以使用异步方法：
const batchScrapeJob = await app.asyncBulkScrapeUrls(['https://docs.firecrawl.dev', 'https://docs.firecrawl.dev/sdks/overview'], {
formats: ['extract'],
extract: {
prompt: '从页面中提取标题和描述。',
schema: schema,
},
});
console.log(batchScrapeJob);
// (异步) 然后可以使用作业ID检查批量抓取的状态：
const batchScrapeStatus = await app.checkBatchScrapeStatus(batchScrapeJob.id);
console.log(batchScrapeStatus);
curl -X POST https://api.firecrawl.dev/v1/batch/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{
"urls": ["https://docs.firecrawl.dev", "https://docs.firecrawl.dev/sdks/overview"],
"formats" : ["extract"],
"prompt": "从页面中提取标题和描述。",
"schema": {
"type": "object",
"properties": {
"title": {
"type": "string"
},
"description": {
"type": "string"
}
},
"required": [
"title",
"description"
]
}
}'
响应
同步
Completed
{
"status": "completed",
"total": 36,
"completed": 36,
"creditsUsed": 36,
"expiresAt": "2024-00-00T00:00:00.000Z",
"next": "https://api.firecrawl.dev/v1/batch/scrape/123-456-789?skip=26",
"data": [
{
"extract": {
"title": "使用Groq Llama 3构建'与网站聊天' | Firecrawl",
"description": "学习如何使用Firecrawl、Groq Llama 3和Langchain构建一个'与您的网站聊天'机器人。"
}
},
...
]
}
异步
{
"success": true,
"id": "123-456-789",
"url": "https://api.firecrawl.dev/v1/batch/scrape/123-456-789"
}