> ## Documentation Index
> Fetch the complete documentation index at: https://firecrawl.web3doc.top/llms.txt
> Use this file to discover all available pages before exploring further.

# 抓取

> 将任何URL转换为干净的数据

## 使用Firecrawl进行抓取

Firecrawl将网页转换为Markdown，非常适合LLM应用程序。原因如下：

1. **管理复杂性：**
   处理代理、缓存、速率限制和JavaScript阻止的内容，以实现顺畅的抓取。

2. **动态内容：**
   从JavaScript渲染的网站、pdf、图片等中收集数据。

3. **Markdown或结构化数据转换：**
   将收集到的数据转换为干净的Markdown或结构化输出，非常适合LLM处理或其他任务。

更多详情，请参阅[抓取端点API参考](https://docs.firecrawl.dev/api-reference/endpoint/scrape)。

## 抓取一个URL

### /scrape端点

用于抓取URL并获取其内容。

### 安装

<CodeGroup>
  ```bash Python theme={null}
  pip install firecrawl-py
  ```

  ```bash JavaScript theme={null}
  npm install @mendable/firecrawl-js
  ```

  ```go Go theme={null}
  go get github.com/mendableai/firecrawl-go
  ```

  ```toml Rust theme={null}
  # Add this to your Cargo.toml
  [dependencies]
  firecrawl = "^0.1"
  tokio = { version = "^1", features = ["full"] }
  serde = { version = "^1.0", features = ["derive"] }
  serde_json = "^1.0"
  uuid = { version = "^1.10", features = ["v4"] }

  [build-dependencies]
  tokio = { version = "1", features = ["full"] }
  ```
</CodeGroup>

### 使用示例

<CodeGroup>
  ```python Python theme={null}
  from firecrawl import FirecrawlApp

  app = FirecrawlApp(api_key="YOUR_API_KEY")

  content = app.scrape_url("https://mendable.ai")
  ```

  ```JavaScript JavaScript theme={null}
  import { FirecrawlApp } from 'firecrawl-js';

  const app = new FirecrawlApp({ apiKey: 'YOUR_API_KEY' });

  const content = await app.scrapeUrl('https://mendable.ai');
  ```

  ```go Go theme={null}
  import (
    "fmt"
    "log"

    "github.com/mendableai/firecrawl-go"
  )

  func main() {
    app, err := firecrawl.NewFirecrawlApp("YOUR_API_KEY")
    if err != nil {
      log.Fatalf("Failed to initialize FirecrawlApp: %v", err)
    }

    content, err := app.ScrapeURL("mendable.ai", nil)
    if err != nil {
      log.Fatalf("Failed to scrape URL: %v", err)
    }
    fmt.Println(content)
  }
  ```

  ```rust Rust theme={null}
  use firecrawl::FirecrawlApp;

  #[tokio::main]
  async fn main() {
      // 用API密钥初始化FirecrawlApp
      let api_key = "YOUR_API_KEY";
      let api_url = "https://api.firecrawl.dev";
      let app = FirecrawlApp::new(api_key, api_url).expect("Failed to initialize FirecrawlApp");

      // 抓取URL
      let scrape_result = app.scrape_url("https://mendable.ai", None).await;

      // 打印抓取结果
    match scrape_result {
      Ok(data) => println!("Scrape Result:
  {}", data["markdown"]),
      Err(e) => eprintln!("Scrape failed: {}", e),
    }
  }
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.firecrawl.dev/v0/scrape \
      -H 'Content-Type: application/json' \
      -H 'Authorization: Bearer YOUR_API_KEY' \
      -d '{
        "url": "https://mendable.ai"
      }'
  ```
</CodeGroup>

### 响应

SDK将直接返回数据对象。cURL将返回如下所示的负载。

```json theme={null}
{
  "success": true,
  "data": {
    "content": "原始内容 ",
    "markdown": "# Markdown内容",
    "provider": "web-scraper",
    "metadata": {
      "title": "Mendable | AI for CX and Sales",
      "description": "AI for CX and Sales",
      "language": null,
      "sourceURL": "https://www.mendable.ai/"
    }
  }
}
```