> ## Documentation Index
> Fetch the complete documentation index at: https://firecrawl.web3doc.top/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl

> Firecrawl can recursively search through a urls subdomains, and gather the content

Firecrawl thoroughly crawls websites, ensuring comprehensive data extraction while bypassing any web blocker mechanisms. Here's how it works:

1. **URL Analysis:**
   Begins with a specified URL, identifying links by looking at the sitemap and then crawling the website. If no sitemap is found, it will crawl the website following the links.

2. **Recursive Traversal:**
   Recursively follows each link to uncover all subpages.

3. **Content Scraping:**
   Gathers content from every visited page while handling any complexities like JavaScript rendering or rate limits.

4. **Result Compilation:**
   Converts collected data into clean markdown or structured output, perfect for LLM processing or any other task.

This method guarantees an exhaustive crawl and data collection from any starting URL.

## Crawling

### /crawl endpoint

用于爬取一个URL及其所有可访问的子页面。这会提交一个爬虫任务并返回一个作业ID，以检查爬虫的状态。

### 安装

<CodeGroup>
  ```bash Python theme={null}
  pip install firecrawl-py
  ```

  ```bash JavaScript theme={null}
  npm install @mendable/firecrawl-js
  ```

  ```go Go theme={null}
  go get github.com/mendableai/firecrawl-go
  ```

  ```toml Rust theme={null}
  # add the following to your Cargo.toml

  [dependencies]
  firecrawl = "^0.1"
  tokio = { version = "^1", features = ["full"] }
  serde = { version = "^1.0", features = ["derive"] }
  serde_json = "^1.0"
  uuid = { version = "^1.10", features = ["v4"] }

  [build-dependencies]
  tokio = { version = "1", features = ["full"] }
  ```
</CodeGroup>

### 使用

<CodeGroup>
  ```python Python theme={null}
  from firecrawl import FirecrawlApp

  app = FirecrawlApp(api_key="YOUR_API_KEY")

  crawl_result = app.crawl_url('mendable.ai', {'crawlerOptions': {'excludes': ['blog/*']}})

  # Get the markdown
  for result in crawl_result:
      print(result['markdown'])
  ```

  ```js JavaScript theme={null}
  import FirecrawlApp from '@mendable/firecrawl-js';

  // Initialize the FirecrawlApp with your API key
  const app = new FirecrawlApp({ apiKey: 'YOUR_API_KEY' });

  // Crawl a website
  const crawlResult = await app.crawlUrl('mendable.ai', { crawlerOptions: { excludes: ['blog/*'] } });

  // Log the markdown
  console.log(crawlResult.map(result => result.markdown));
  ```

  ```go Go theme={null}
  import (
    "fmt"
    "log"

    "github.com/mendableai/firecrawl-go"
  )

  func main() {
    app, err := firecrawl.NewFirecrawlApp("YOUR_API_KEY")
    if err != nil {
      log.Fatalf("Failed to initialize FirecrawlApp: %v", err)
    }

    crawlResult, err := app.CrawlURL("mendable.ai", nil)
    if err != nil {
      log.Fatalf("Failed to crawl URL: %v", err)
    }

    fmt.Println(crawlResult)
  }
  ```

  ```rust Rust theme={null}
  use firecrawl::FirecrawlApp;

  #[tokio::main]
  async fn main() {
    // Initialize the FirecrawlApp with the API key
    let api_key = "YOUR_API_KEY";
    let api_url = "https://api.firecrawl.dev";
    let app = FirecrawlApp::new(api_key, api_url).expect("Failed to initialize FirecrawlApp");

    // Crawl the URL
    let crawl_params = json!({
      "crawlerOptions": {
          "excludes": ["blog/*"]
      }
    });

    let crawl_result = app
        .crawl_url("https://example.com", Some(crawl_params), true, 2, None)
        .await;

    // Print the crawl result
    match crawl_result {
        Ok(data) => println!("Crawl Result:
  {}", data),
        Err(e) => eprintln!("Crawl failed: {}", e),
    }
  }
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.firecrawl.dev/v0/crawl \
      -H 'Content-Type: application/json' \
      -H 'Authorization: Bearer YOUR_API_KEY' \
      -d '{
        "url": "https://mendable.ai"
      }'
  ```
</CodeGroup>

### Job ID Response

如果您不使用SDK或更喜欢使用webhook或其他轮询方法，可以将`wait_until_done`设置为`false`。这将返回一个jobId。
对于cURL，/crawl将始终返回一个jobId，您可以用它来检查爬虫的状态。

```json theme={null}
{ "jobId": "1234-5678-9101" }
```

### 检查爬虫作业状态

用于检查爬虫作业的状态并获取其结果。

<CodeGroup>
  ```python Python theme={null}
  status = app.check_crawl_status(job_id)
  ```

  ```js JavaScript theme={null}
  const status = await app.checkCrawlStatus(jobId);
  ```

  ```go Go theme={null}
  status, err := app.CheckCrawlStatus(jobId)
  if err != nil {
    log.Fatalf("Failed to check crawl status: %v", err)
  }
  ```

  ```rust Rust theme={null}
  let status = match app.check_crawl_status(jobId).await {
      Ok(status) => status,
      Err(e) => panic!("Failed to check crawl status: {:?}", e),
  };

  println!("Crawl Status: {:?}", status);
  ```

  ```bash cURL theme={null}
  curl -X GET https://api.firecrawl.dev/v0/crawl/status/1234-5678-9101 \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY'
  ```
</CodeGroup>

#### 响应

```json theme={null}
{
  "status": "completed",
  "current": 22,
  "total": 22,
  "data": [
    {
      "content": "Raw Content ",
      "markdown": "# Markdown Content",
      "provider": "web-scraper",
      "metadata": {
        "title": "Mendable | AI for CX and Sales",
        "description": "AI for CX and Sales",
        "language": null,
        "sourceURL": "https://www.mendable.ai/"
      }
    }
  ]
}
```