使用Firecrawl搜索网页并抓取结果
Firecrawl将其SERP（搜索引擎结果页面）API与其强大的抓取基础设施集成，通过单一端点提供无缝的搜索和抓取功能。原因如下：
-
统一的搜索查询：
用户通过SERP端点提交搜索查询。
-
自动化结果抓取：
Firecrawl自动处理搜索结果，并利用其抓取能力从每个结果页面提取数据。
-
数据交付：
从所有结果页面抓取的数据被编译并以干净的markdown格式交付，随时可用。
这种集成使用户能够高效地执行网络搜索，并以最少的努力从多个来源获得全面、抓取的数据。
有关更多详细信息，请参阅搜索端点文档。
搜索任何查询
/search 端点
用于搜索网络，获取最相关的结果，抓取每个页面并返回markdown。
curl -X POST https://api.firecrawl.dev/v0/search \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{
"query": "firecrawl",
"pageOptions": {
"fetchPageContent": true // false for a fast serp api
}
}'
{
"success": true,
"data": [
{
"url": "https://docs.firecrawl.dev",
"markdown": "# Markdown Content",
"provider": "web-scraper",
"metadata": {
"title": "Firecrawl | Scrape the web reliably for your LLMs",
"description": "AI for CX and Sales",
"language": null,
"sourceURL": "https://docs.firecrawl.dev/"
}
}
]
}
使用Python SDK
安装Python SDK
搜索查询
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key="YOUR_API_KEY")
result = app.search(query="What is firecrawl?")
响应将与上面的curl命令中显示的内容相似。
使用JavaScript SDK
安装JavaScript SDK
npm install @mendable/firecrawl-js
搜索查询
import FirecrawlApp from '@mendable/firecrawl-js';
// 使用您的API密钥初始化FirecrawlApp
const app = new FirecrawlApp({ apiKey: 'YOUR_API_KEY' });
// 执行搜索
const result = await app.search('What is firecrawl?');
响应将与上面的curl命令中显示的内容相似。
使用Go SDK
安装Go SDK
go get github.com/mendableai/firecrawl-go
搜索查询
import (
"fmt"
"log"
"github.com/mendableai/firecrawl-go"
)
func main() {
app, err := firecrawl.NewFirecrawlApp("YOUR_API_KEY")
if err != nil {
log.Fatalf("Failed to initialize FirecrawlApp: %v", err)
}
query := "What is firecrawl?"
searchResult, err := app.Search(query)
if err != nil {
log.Fatalf("Failed to search: %v", err)
}
fmt.Println(searchResult)
}
使用Rust SDK
安装Rust SDK
在您的Cargo.toml中添加以下内容：
[dependencies]
firecrawl = "^0.1"
tokio = { version = "^1", features = ["full"] }
serde = { version = "^1.0", features = ["derive"] }
serde_json = "^1.0"
uuid = { version = "^1.10", features = ["v4"] }
[build-dependencies]
tokio = { version = "1", features = ["full"] }
搜索查询
async fn main() {
let api_key = "YOUR_API_KEY";
let api_url = "https://api.firecrawl.dev";
let app = FirecrawlApp::new(api_key, api_url).expect("Failed to initialize FirecrawlApp");
let query = "What is firecrawl?";
let search_result = app.search(query).await;
match search_result {
Ok(data) => println!("Search Result: {}", data),
Err(e) => eprintln!("Failed to search: {}", e),
}
}
